Ramesh said such people should be asked to "go home".
The minister said he felt "diminished as a human being" after the December incident which has hurt India's image across the world.
"None of us have come out of this looking good. Personally, as a man, as an Indian, I felt demeaned by what happened. I felt diminished as a human being," he said.
Asked whether those who made insensitive statements should be punished or not, Ramesh said, "Privately, I think anybody who makes such comments should be asked to go home. You make this comment with full knowledge. It is not that you have been misquoted... I find it appalling actually."
Justifying the angry reaction from masses after the incident, he said the opportunity should be used to sensitise people and initiate political reforms in the country.
In a letter to Delhi Chief Minister Sheila Dikshit, the Rural Development Minister has suggested that the women helpline (181) should only have women telephone operators so that the caller can convey her problem without any hesitation.
Student volunteers could also be considered for the post of telephone operators, he said.
"On the helpline, I have been told that in one instance a male telephone operator who took the call was asking woman caller about her marital and living status.
"To ensure, these instances don't happen and women feel safe and comfortable in registering their complaints, the helpline should have only women telephone operators. Student volunteers could also be considered," Ramesh said.
(With inputs from PTI)