Sonia Gandhi "wasn't very fond of him", the former external affairs minister and friend of the Gandhi family says in "One Life is Not Enough" (Rupa).
Rao, who was prime minister for five years until 1996, was aware of Gandhi's attitude, the 410-page book says.
Natwar Singh says that Rao "seemed uncharacteristically agitated and restless" when they met one day in December 1994. The book quotes Rao as telling Natwar Singh: "I can take on Sonia Gandhi. But I do not want to do so. Some of her advisers have been filling her ears against me. I don't take them seriously. Sonia's case is different.
"Her attitude towards me is affecting my health. If she wants me to go, she only has to say so.
"I have done my best to meet all her desires and requirements promptly. You worked closely with her and must know why Sonia is hostile to me," he said.
Rao was reportedly so eager to get into Gandhi's good books that he had "a clandestine rendezvous" with Mohammad Younus, a friend of the Gandhis at Natwar Singh's suggestion to reach out to her.
"That P.V. (Narasimha Rao) agreed to such a meeting was in itself a clear indication of how keen he was for a patch-up with Sonia. But it did not happen."
Such was the Gandhi family's antipathy to Rao, that after he died in 2004, his body was sent to his native Hyderabad for the last rites. He is the only former prime minister who does not have a memorial in Delhi.
In the book, portions of which have made headline news in recent days, Natwar Singh says it was Rahul Gandhi who prevented Sonia Gandhi, his mother, from becoming the prime minister in 2004 after the Congress-led UPA won the Lok Sabha election.
"Rahul was vehemently opposed to his mother becoming prime minister, fearing that she would lose her life, much like his grandmother and his father.
"Matters reached a climax after Rahul said he was prepared to take any possible step to prevent his mother from taking up the prime ministership.
"Rahul is a strong-willed person; this was no ordinary threat. He gave Sonia 24 hours to decide. Manmohan Singh, Suman Dubey, Priyanka and I were present at that moment.
"Sonia was visibly agonized and in tears. As a mother, it was impossible for her to ignore Rahul. He had his way. That was the reason for her not becoming prime minister.
"Only Manmohan Singh and I were aware of Sonia's decision."
Natwar Singh says once the Congress-led UPA took power, "it was widely known that Sonia very discreetly monitored the functioning of the most important ministries in the government, displaying a Machiavellian side to her character".
Natwar Singh, who quit the Congress after being accused of improper conduct after his son's alleged involvement in a UN oil-for-food deal with Iraq, says Sonia Gandhi's image was not flattering. "To an extent, she has herself to blame for it.
"She never lets her guard down, never gives away what is in her mind. Obsessively secretive and suspicious, she evokes awe, not admiration."
The book says: "Over the years she has evolved from being a diffident, nervous, shy woman to an ambitious, authoritarian and stern leader. Her displeasure strikes fear among Congressmen...
"Under her, dissent is smothered, free discussion fenced in. Silence is used as a weapon and every subtle gesture is a message, an icy stare a warning...
"Favours are granted piecemeal, lese-majeste is given short shrift. Beneath all that posturing an ordinary and insecure person emerges. Her capriciousness is lauded. A fine-tuned personality cult is promoted. Politics has coarsened her."