"The world has changed and the UN must change along with it. The UN Security Council has to be reformed," Annan said at a panel discussion at Oxford University on Thursday.
"Whether or not a country has veto power in the UN Security Council, its views must be respected," Annan added.
Carter also voiced his support for the UN reform, proposing an increase of the authority of the UN General Assembly.
He said that a good example of the value of the UN General Assembly was when it voted to recognise Palestine as a non-member observer state.
In 2012, the UN General Assembly voted overwhelmingly to grant an upgrade of Palestine's status from "entity" to "non-member state", an implicit and symbolic recognition of Palestine's statehood at the UN.
Annan and Carter were joined by former president of Ireland Mary Robinson and former world leaders from the Elders, an independent group of senior world leaders who work together for peace and human rights, in the panel discussion.
The Elders was brought together in 2007 by Nelson Mandela and is now chaired by Annan.
The discussion focused mainly on 21st century peace building, conflict resolution, good governance and women's role in world politics.