President during Afghanistan's bloody 1992-96 civil war and a warlord with a chequered human rights record, the 71-year-old Rabbani was killed at home by a Taliban suicide bomber wearing explosives in his turban, police said.
Although there has been no official word from the Taliban, his killing deals a heavy blow to already remote hopes of an imminent end to 10 years of fighting between Islamist insurgents and the Afghan government backed by Western troops.
President Hamid Karzai was today rushing back to Afghanistan, cutting short a visit to the UN General Assembly in New York after meeting US President Barack Obama as preparations were being made for Rabbani's funeral.
It was arguably the most high-profile political assassination since the 2001 US-led invasion dislodged the Taliban, two months after President Hamid Karzai's younger brother was killed in the Taliban's southern heartland.
Head of Karzai's High Council for Peace for 11 months, Rabbani fronted the government's attempts to broker peace with the Taliban, but his efforts so far appeared to come to little despite growing US interest in a settlement.
Nevertheless the assassination of such a high-profile Karzai ally, at his home close to the US embassy in allegedly the most protected part of the capital, underscores the sharp deterioration in security in Afghanistan.
Regardless of whether Rabbani was even capable of brokering peace in a country ripped apart by three decades of war, his killing is a deep symbolic blow to the government's efforts and highlights the perceived strength of the insurgency.
As a Tajik, Rabbani's killing may also further upset the delicate and at times increasingly fraught relations between Afghanistan's different minority ethnic groups and the dominant Pashtun community.
Today, several hundred people carrying pictures of Rabbani and banners gathered near his house to remember him and protest against his killing, an AFP photographer said.
Many wore black headbands in mourning. Government forces stepped up security in the diplomatic zone where Rabbani's house is located, with cars barred from entering the area and police stopping and searching many people on foot.
People gathered at the gate of Rabbani's house to sit and recite the Koran. A stream of government officials also attended to pay their respects.