Modi was responding to Prime Minister Manmohan Singh's earlier remarks that Sardar Patel was a Congressman.
Speaking at an event after Bharatiya Janata Party (BJP) leader L.K. Advani laid the foundation stone of the proposed 182-metre-tall statue of Sardar Patel here, Modi attacked the prime minister over "delay" in permission for erecting gates of the Sardar Sarovar Dam.
"To link Sardar Patel with any party will be a big injustice to him (Patel). That party (Congress) was a part of his life and both history and Modi cannot deny it," the BJP's prime ministerial candidate said.
Hitting out at Congress accusations that he (Modi) was trying to usurp Sardar Patel's legacy, Modi said leaders such as Rana Pratap, Bhagat Singh and Shivaji were not from the BJP, but he respected them all.
After Modi said Tuesday that he wished Sardar Patel had been India's prime minister, Manmohan Singh responded that the "Iron Man" had always been a Congressman, and secular too.
Modi said he appreciated the prime minister's remarks that Sardar Patel was truly secular.
"We also say that this country needs the secularism of Sardar Patel, not vote bank secularism," Modi said.
"He (Sardar Patel) was secular but that didn't come in the way of his rebuilding the Somanth temple. He (Sardar Patel) moved ahead, keeping in mind the country's heritage. The country needs the secularism of Sardar Patel. It will never be divided," Modi said.
Modi said though Mahatma Gandhi fought to free the country of untouchability, "political untouchability" still exists. "We should also try and remove political untouchability," he said.
The BJP leader alleged that despite repeated requests to Prime Minister Manmohan Singh, permission had not been granted to erect gates of the Sardar Sarovar Dam.
He said only the work of erecting gates is pending; more water will be filled in the dam if that task was accomplished.
"I have to say with great pain... I have met the prime minister several times. I told him that it will take us three years to do the work. Let us erect gates. If there is an issue with permission to close them, take a call on the issue later," Modi said.
Modi said the prime minister acknowledges that his suggestion is fine.
"When I meet him again, he says, 'Is it that the work has not been done?'. When I meet him after one year, he says, 'Is it that the work has not been done?'. Now tell me, what should I do?" Modi asked the gathering.
Modi said he appreciated the PM's remarks that Sardar Patel was truly secular.
Modi also said there were some "friends" who had taken a "supari (undertaking)" to finish him, and were "resorting to spreading dirt".
Referring to the foundation stone of Sardar Patel's proposed statue, Modi said "it is the beginning of a new resolve".
In his speech, Advani praised Modi, who has been chosen the BJP's prime ministerial candidate.
He said Patel, the "Iron Man", was responsible for the unity and the integration of princely states following the country's Independence in 1947.
"That is why, if Modi has pledged to build a huge statue (of Sardar Patel), which will be bigger than the Statue of Liberty, it makes us very happy and proud," Advani said.
"I compliment Modi, whom our party has chosen as the prime ministerial candidate. I want to share my pride and happiness (with all of you)," Advani said. "When Britain split India in 1947, 565 small and big princely states were given back their sovereignty. I believe that if there was not a leader like Sardar (Patel), then India would not just have been divided, but disintegrated."
The proposed statue of Sardar Patel will be the tallest statue in the world, Modi said. It will be double the height of the Statue of Liberty in the US.