Khan had said on Thursday that Taj Mahal should be handed over to the Waqf board. Another Muslim leader has asked the Samajwadi Party to allow Muslims to offer prayers five times a day at the Taj Mahal.
The residents of Agra are fuming over the remarks.
"He (Khan) needs to get himself checked for mental balance," said Surendra Sharma, president of the Braj Mandal Heritage Conservation Society. Sharma said Khan should stay away from making such comments.
More than eight million people visit the 17th century Taj Mahal annually.
Ajmer-based Mughal historian R. Nath said: "The Taj is a climax of 5,000 years of Indian excellence. I am terribly saddened by the minister's statement."
Local tourism organisations have protested the minister's statement and Hindu groups such as Vishwa Hindu Parishad and the Bajrang Dal are planning protests and demonstrations.
BJP legislator Yogendra Upadhaya said Khan should resist making such absurd statements. "We will vehemently oppose any such move," he said.
Khan had earlier, in June 2013, said that he would have led mob to demolish Taj Mahal. (Read here)