The vote, which required a simple majority, ended with 180 in favor of ousting Rajoy, 169 against and one abstention.
Minutes after narrowly winning a no-confidence vote in parliament, the Socialist party leader signalled a change in tone and priorities from Rajoy's unbending commitment to reducing the national debt during his more than six years as prime minister.
The reputation of Rajoy's Popular Party's was badly damaged by a court verdict last week that identified it as a beneficiary of a large kickbacks-for-contracts scheme.
The 46-year-old leader of the Spanish Socialist Workers' Party came to power at a ceremony at the Zarzuela Palace on the outskirts of Madrid, presided over by King Felipe VI.
He made brief farewell remarks to lawmakers before the vote, telling them that "it has been an honour to leave Spain better than I found it".
Socialist leader Pedro Sanchez in the parliament after the motion of no confidence vote.
"Our "yes" to Sanchez is a "no" to Rajoy", Mr Tarda said.
Rajoy went to shake hands with Sanchez after the result was announced.
He was ousted by his own party's heavyweights in 2016 over back-to-back losses in general elections and after he tried to block Rajoy's bid to form a government.
"I am aware of the responsibility I am assuming, of the complex political moment our country is going through and I will rise to all the challenges with humility and dedication", Sanchez said after the vote to remove Rajoy from office.
The former economics professor and career politician inherits a strong economy in which growth previous year reached 3.1 percent.
Although he is now prime minister, Sánchez's party commands less than 25% of seats in parliament.
Podemos has already asked to be part of his new government.
In its ruling, the court said the credibility of Rajoy's testimony "should be questioned".
The no-confidence motion was initiated by the opposition Socialist Workers' Party (PSOE) on Thursday and supported by Catalan parties and the leftist Podemos.
His failure to resolve the political crisis caused by the standoff contributed to dissatisfaction with him, though the no-confidence vote - backed by Catalan pro-independence parties - was triggered primarily by a long-running corruption trial involving members of his center-right party.
Mr Sanchez indicated during the debate on the motion that he would try to govern until the scheduled end of the parliamentary term in mid-2020.
He has also said he wants to "build bridges" with Catalonia's new separatist government, headed by Quim Torra, which will take office on Saturday at the same time that Sanchez takes his oath of office.
The members of Catalonia's former government are to be tried on charges including rebellion.