Manmohan Singh also told the media that "private conversations should not be misused for capital gains".
Congress chief Sonia Gandhi, earlier in the day, refuted the allegations made by Natwar Singh in his book and said she would write her own book to tell the truth.
Natwar Singh, a former minister in the UPA government, alleged in an interview that Sonia Gandhi's decision not to take up the post of prime minister in 2004 was not because of an "inner voice" as she had famously said.
He said it was because of opposition from her son Rahul Gandhi, who was reportedly worried that she would be assassinated like his father Rajiv Gandhi and grandmother Indira Gandhi, both of whom were prime ministers.
He also claimed that Sonia Gandhi had access to government files and these were taken to her by bureaucrat Pulok Chatterji.
Natwar Singh, 83, had to resign from the Congress-led coalition government in 2005 after allegations of corruption.