Speaking on the occasion, the President recalled that South Africa's anti-apartheid icon Nelson Mandela was also imprisoned in this old fort for some time.
The bust has been sculpted by Gautam Pal and presented by the ICCR to the Constitution Court, which the prison has now been converted.
Writing in 1994, to mark the 125th birth anniversary of Gandhiji, which was also the year, freedom and democracy dawned in South Africa, Mandela describing his affinity with Gandhiji wrote, "Though separated in time, there remains a bond between us, in our shared prison experiences, our defiance of unjust laws and in the fact that violence threatens our aspirations for peace and reconciliation", the President said.
The President went round a Museum on the Mahatma. The great granddaughter of Mahatma Gandhi, Kirti Meon, who is the Registrar in a University here was also present on the occasion.
With South Africa transforming emblems of its repressive past into custodians of democracy and freedom, the notorious Old Fort Prison, a high security jail built in 1890, has been converted into Constitution Hill, housing the highest court of the country, the Constitution Court.
Chief Justice of South Africa, Mogoeng Mogoeng, who also participated in the function paid glowing tributes to the Mahatma.
The President, who will be going round other places associated with Mahatma, said, "South Africa is the place where the world witnessed for the first time Gandhi's contribution to political transformation through non-violence and peaceful dialogue".
She said the people of India owe a deep sense of gratitude to South Africa for giving us the Mahatma.
"During the 21 years of his stay in South Africa, he shaped his philosophy of 'Satyagraha and Non-Violence', which later became potent tools for liberation of the people of India from the yoke of colonialism".
She hoped that the Gandhiji's bust will symbolise the values that are enshrined in the histories of India and South Africa, and underpin their continuing resolve to fight injustice and inequality in the world.
Gandhi's great granddaughter Kirti Menon said the bust that represents 'Barrister Gandhi' would help to spread the Mahatma's message of peace and non-violence that is relevant today.
She said both India and South Africa are facing the same type of problems and are working to address issues of poverty and ensuring that everyone has access to health and education.
Menon said as a South African she loves India but her home was South Africa, and that the Gandhi committee here was working for the ideals of the Mahatma.
In South Africa, Patil will also visit other places associated with Gandhi, including Pietermaritzburg Station where he was thrown out of a first class compartment of a train during the apartheid regime.
In Durban, Patil will visit the Phoenix Settlement and offer floral tributes at the bust of Gandhi.
The President is also expected to visit Robben Island where Mandela was imprisoned.