External Affairs Minister G.L. Peiris said that there was no reason for people to be "obsessed" over why someone is not attending the summit, Xinhua reported.
Addressing a press briefing, he said that Sri Lanka rejoices over the friends it has in the Commonwealth who are attending the summit.
Indian Prime Minister Manmohan Singh not likely to attend.
Side events related to the summit are already underway, while the heads of government meeting takes place on Friday.
Heads of government from Canada and India are among those boycotting the summit over human rights issue and domestic pressure, while Mauritius announced its prime minister will also not attend.
Peiris said that the Sri Lankan government has done its part to invite the Commonwealth leaders for the summit and it is up to those invited to decide if they want to attend or not.
Asked if Indian Prime Minister Manmohan Singh had given any reason as to why he was not attending the summit, Peiris said that no country needs to explain why it is not attending an event.
He also noted that the Commonwealth is a voluntary association of sovereign states and each country must respect the identity of the other.
Sri Lanka has been accused of committing grave human rights abuses during the final stage of the war against the Tamil Tiger rebels in the north of the country, an allegation the government denies.