In the book that hit the stands Friday, Baru, who now works with a think tank, said so weak was Manmohan Singh that he also failed to defend his (Baru's) promised reappointment after the 2009 election victory as secretary in the PMO and tamely surrendered to the party's veto.
"To tell the truth, I was dismayed by the PM's display of spinelessness," he said.
Baru, who spoke to Times Now news channel in the evening, also said there had been weakening of the office of prime minister in the coalition era.
Baru said the Congress never acknowledged Manmohan's effort in forming UPA II
He also said that it was widely known in the PMO that some decisions taken by the prime minister went through a pre-verification process to get Gandhi's verbal nod and this was done by the prime minister's principal secretary Pulok Chatterji, who was once her close aide.
"He (Chatterji) was very much part of the family. I would not say he took files to her physically because I am not witness to it. But we knew there were a lot of issues (he) would get her approval for before the prime minister could finally agree. I think lots of us in the PMO knew that," Baru said.
In his book, Baru gave instances of Manmohan Singh capitulating to the extra-constitutional authority of Gandhi and refusing to assert himself.
"Initially, I saw his subservience as an aspect of his shy and self-effacing personality, but over time I felt, like many, that this might be his strategy for political survival. "Was it just unquestioned loyalty to the leader or a survival instinct that prompted him to remain? Whatever the motive, his image took a fatal blow," Baru said in the book "The Accidental Prime Minister: The Making and Unmaking of Manmohan Singh".
"So I, like millions of his middle-class supporters, feel tragically cheated that he has allowed himself to become an object of such ridicule in his second term in office, in the process devaluing the office of the prime minister," he wrote in the book.
The BJP promptly reacted with a I-told-you-so, with party leader M Venkaiah Naidu saying that "I have been saying from day one, PM presides, Madam decides."
The Congress, battling countrywide negative perceptions about the government and widely expected to do badly in these polls, was left red-faced.
Baru, during whose tenure in the first term of Manmohan Singh (2004-2009) enjoyed a positive media, was always uncomfortable with the Congress establishment who made no secret of the fact that they wanted him replaced in Manmohan Singh's second term.
Pankaj Pachauri, who is presently the communication adviser in the Prime Minister's Office, put out a statement saying: "It is an attempt to misuse a privileged position and access to high office to gain credibility and to apparently exploit it for commercial gain.
"The commentary smacks of fiction and coloured views of a former adviser. The question about comments of the former media adviser was raised by senior editors when they met the prime minister in October last year. His answer was 'Do not believe all he is saying'."
Commenting on the PMO's statement during his interview on Times Now news channel, Baru said it could have been better drafted and added that to describe it as fiction was "ridiculous."
He contended the book was a balanced account of UPA-I government.
"It was success of UPA-I that made UPA-II possible," he said.
Baru also said that he had held back what was told to him in confidence.
"(I) have held back quite a lot," Baru said.