The biggest shakeup of Trump's Cabinet since he took office more than a year ago, which the president announced on Twitter, comes as the administration prepares for an unprecedented meeting with the leader of North Korea.
Trump tapped the CIA's deputy director, Gina Haspel, to replace Pompeo at the intelligence agency.
Tillerson's departure caps months of tensions between the Republican president and the 65-year-old former Exxon Mobil chief executive.
Tillerson was not officially notified about his dismissal prior to the president's Tuesday morning tweet, according to CNN.
On Monday Tillerson blamed Russia for the poisonings in England of a former Russian double agent and his daughter. Earlier, White House press secretary Sarah Sanders refrained from saying Moscow was responsible.
Tillerson was reported to have privately called Trump a "moron" in July after the president suggested a 10-fold increase in the US nuclear arsenal. Tillerson declined to directly address whether he had made the comment, although a State Department spokeswoman later issued a denial.
A senior White House official said Trump asked Tillerson to step down on Friday but did not want to make it public while he was on a trip to Africa. Trump's announcement came only a few hours after Tillerson landed in Washington after a trip that had been cut short.
The official said Trump works well with Pompeo, a former Republican congressman from Kansas, and wanted him in place before the US president's planned talks with North Korean leader Kim Jong Un and trade negotiations.
Tillerson had no diplomatic or political experience before becoming secretary of state. He appeared out of the loop last week when Trump announced he would meet with North Korea's Kim.
U.S. stock index futures pared their gains and the dollar also trimmed gains versus the yen while extending losses versus the euro amid the news.
"Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job! Thank you to Rex Tillerson for his service! Gina Haspel will become the new Director of the CIA, and the first woman so chosen. Congratulations to all!" Trump said on Twitter.
Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job! Thank you to Rex Tillerson for his service! Gina Haspel will become the new Director of the CIA, and the first woman so chosen. Congratulations to all!— Donald J. Trump (@realDonaldTrump) March 13, 2018
Tillerson joined a long list of senior officials who have either resigned or been fired since Trump took office in January 2017. Others include strategist Steve Bannon, national security adviser Michael Flynn, FBI Director James Comey, White House chief of staff Reince Priebus, health secretary Tom Price, communications directors Hope Hicks and Anthony Scaramucci, economic adviser Gary Cohn and press secretary Sean Spicer.