"The Iranian diplomat was killed on the spot in a terrorist attack on Saturday," reported Xinhua citing Yemen's news agency of Saba.
Yemeni Foreign Minister Abu Bakr al-Qirbi condemned the attack, saying it targeted the Yemen-Iran relationship.
Meanwhile, the country's security sources said the Iranian diplomat died of wounds in hospital as the gunmen fired more than 10 shots against him.
Unidentified gunmen shot at a car belonging to the Iranian embassy near the Iranian ambassador's residence in downtown Sana'a, a security official in the Yemeni interior ministry said.
"The diplomat who is in charge of finance at the Iranian embassy in Sanaa was shot on Saturday afternoon by gunmen driving a car near a supermarket on the Hadda street in downtown the capital, about 100 metres north of the Iranian embassy," the official said requesting anonymity.
The gunmen fled the scene after firing more than 10 shots at the diplomat on the busy street, he added.
No group immediately claimed responsibility for the assassination.