The rich could drive cars to reach their destinations or get their wards admitted to better schools but the poor who depend on public transport, schools and hospitals should be taken care of by the government.
"The government at the Centre is committed to addressing their (poor people's) needs," he said.
Without naming the previous Congress governments at the Centre, Modi said they had nationalised banks four decades ago saying that the money in them would help the poor but it never happened.
Referring to the Jan Dhan Yojana launched by his government, the Prime Minister appreciated the poor for contributing Rs 7,000 crore in opening bank accounts under it.
He expressed concern over the migration of workers from Jharkhand to big cities to eke out a living.
Modi said his endeavour was to set up small factories in every village and town so that the youth could get jobs locally and stay with their families.