"Well, it was a surprise. There had been unrest, but nobody expected that Mrs Gandhi would go that far," Singh's daughter Daman Singh quotes him as saying in her book - "Strictly Personal: Manmohan and Gursharan".
The book is based on Daman's conversations with her parents and hours spent in libraries and archives.
When his daughter asked him how the Emergency affected the government servants, Singh replied, "I think there was a lot more emphasis on punctuality, on discipline. So some good things happened. But I think the atmosphere in the whole country was one of fear. There were arbitrary arrests and detentions."
According to Singh, there was a "lot of unrest in the country, particularly due to the way the family planning programme - the sterilisation programme - was implemented in some of the northern states and in Delhi". He felt Sanjay Gandhi was the most important "extra-constitutional authority".
"He had a lot of influence. A lot of people who dealt with things that were of direct interest to him said that they felt the pressure," Singh recounts.
After the Morarji Desai-led Janata Party won a majority and came to power post-Emergency, a number of officers were shunted out but Singh kept his job.
Singh: Sanjay Gandhi was the most important "extra-constitutional authority"
Initially, Singh felt Desai was not quite fond of him.
"When Morarji Desai became prime minister he had been told that I was close to the previous government. So he was quite rude to begin with. But after some time, he became very fond of me. Morarji Desai was fairly balanced, although people misunderstand him as a very rigid man. I think on the surface he was rigid, but he was amenable to persuasion," Singh is quoted as saying.
After being named to head the government in 1991, P V Narasimha Rao wanted Singh to be his finance minister, the book, published by Harper Collins India, says.
Singh recalls he was asleep when P C Alexander rang him up frantically to convey the decision, which, he says, was out of the blue.
According to Singh, Rao's most important role was that he allowed the process of liberalisation and opening up to go ahead, and gave it his full support.
Singh says Rao was first a a little sceptic about the liberalisation idea and had to be persuaded.
"I had to persuade him. I think he was a sceptic to begin with, but later on he was convinced that what we were doing was the right thing to do, that there was no other way out. But he wanted to sanctify the middle path - that we should undertake liberalisation but also take care of the marginalised sections, the poor," recounts Singh.
"He also jokingly told me that if things worked well we would all claim credit, and if things didn't work out well I would be sacked," he says.