TH Mustafa, a former minister, also demanded Gandhi's sister Priyanka Gandhi be made the new party chief.
Addressing reporters here, Mustafa said that Gandhi should quit from his post and if he does not, he should be removed.
"Rahul behaved like a joker and that's the reason why the Congress suffered a major reversal in the Lok Sabha polls. The role of a prime minister is not child's play and the people knew it and handed out the worst defeat to the Congress party.
"He should take responsibility and quit and if not he should be removed and Priyanka Gandhi should be made the new president of the party," he said.
Mustafa held Rahul responsible for the poll defeat and wanted his removal
"His (Rahul's) mad style of working using computer and internet and in the company of a group of CWC members who only praise whatever he does has caused this defeat. It's unfortunate that even A.K.Antony belongs to this group," said Mustafa, a former minister in the K Karunakaran cabinet (1991-95) and five-time legislator, known for openly attacking top leaders of his party.