In a departure statement, the President said he was leaving with sadness for the memorial service to be held in Johannesburg Tuesday.
Around 90 heads of state and governments from across the world will attend the memorial service.
Pranab Mukherjee led an Indian delegation, which included Congress president Sonia Gandhi, commerce and industry minister Anand Sharma, leader of opposition in the Lok Sabha Sushma Swaraj and MPs Sitaram Yechury of the Communist Party of India (Marxist) (CPM) and Satish Chandra Mishra of the Bahujan Samaj Party (BSP) as well as senior government officials.
The President said participation in the memorial service by a delegation representing a cross-section of the political spectrum "reflects the high degree of love and respect which Mandela commanded in India".
"He was a household name and someone, we saw as one of our own. His life was a living example of human strength and courage in the face of brute force and gross injustice.
"He was the last of the giants who led the world's struggles against the colonialism and his struggle held special significance for us as we saw in him a reflection of our own prolonged anti-colonial struggle led by Mahatma Gandhi," Mukherjee said.
"India and South Africa are countries and peoples who have shared both the tragedy of colonialism and the triumph of the free spirit. Today, we stand together united in our determination to realize common goals and build a new world," he said.
He said the delegation hopes to convey to the government and people of South Africa "the intense grief and personal loss that we in India feel over the sad demise of the great soul - our beloved 'Madiba'."
Mandela, who died on December 5 at the age of 95, was a recipient of India's highest civilian award - the Bharat Ratna. He was a believer in the principles of Mahatma Gandhi, who initially served in South Africa and later went on to lead India to freedom.
Mandela served as president of South Africa from 1994 to 1999.
(With IANS inputs)