Mandela passed away peacefully in the company of his family at 8.30 pm Thursday, state news agency sanews.gov.za reported.
In his live televised address to the nation, President Zuma said "this is a moment of deepest sorrow for South Africa as the nation has lost its greatest son".
"Although we knew that this day would come, nothing can diminish our sense of a profound and enduring loss," he said.
Nelson Mandela passed away in Thursday night at the age of 95
Zuma said the nation's thoughts were with Madiba's wife Graca Machel, his former wife Winnie Madikizela-Mandela, his children, his grandchildren, his great grand-children and the entire family as well as his friends, comrades and colleagues.
"Our thoughts are with the South African people who today mourn the loss of the one person who, more than any other, came to embody their sense of a common nationhood. Our thoughts are with the millions of people across the world who embraced Madiba as their own, and who saw his cause as their cause," said the president.
He said that Madiba (as Mandela was called with love by his clan name) had brought us together, and "it is together that we will bid him farewell".
"Let us express, each in our own way, the deep gratitude we feel for a life spent in service of the people of this country and in the cause of humanity. This is indeed the moment of our deepest sorrow. Yet it must also be the moment of our greatest determination," said Zuma.
The president urged South Africans to conduct themselves with dignity and respect as they paid their last respects to the father of the nation.
"Let us be mindful of his wishes and the wishes of his family. As we gather, wherever we are in the country and wherever we are in the world, let us recall the values for which Madiba fought."
Mandela served as president of South Africa from 1994 to 1999. He had battled health problems in recent months, including a recurring lung infection that led to numerous hospitalisations.
He was released from hospital in early September following an 85-day stay for a recurring lung infection, the result of his longtime imprisonment during the apartheid times.
Since he was released from hospital, the South African presidency repeatedly had described Mandela's condition as critical but stable.
Madiba will receive a state funeral and all flags in the country will fly at half-mast Friday and remain so until after the funeral.