In his "Mann ki Baat" radio address, Modi also said he was excited about inaugurating an Ambedkar Bhavan in London next month.
In his nearly half-hour speech, the prime minister did not mention any of the recent incidents of violence including the lynching of a Muslim man in Uttar Pradesh over rumours that he ate beef.
But he underlined that India was a diverse country, "and this diversity is the pride of India... Peace, harmony and unity... these are the key to development".
Modi also recalled Sardar Vallabhbhai Patel, whose birth anniversary falls on October 31, and said he made a huge contribution to India's unity.
"He had a dream for a united India, and he succeeded in doing that geographically."
The prime minister added that the 'mantra' of unity should "constantly be a part of our thought, behaviour and expressions".
Modi said the government had bought the house where B.R. Ambedkar used to live in London.
"I will be making a trip to Britain a day after Diwali. I am looking forward to my trip. I'll inaugurate the Ambedkar Bhavan in London, which is now the Indian government's property," he said.
"For the Dalits, oppressed, poor, who spend their lives with difficulties, Baba Saheb Ambedkar Bhavan will be an inspiration that with will power all difficulties can be overcome," he said.
Modi said governments in India provide scholarships to students from Dalit, tribal and backward communities to study abroad.
"The government of India and state governments give scholarship to bright students who want to go abroad for studies. The Indian government also encourages such Dalit youth," he said.