But he underlined that it would be wrong to "look at any one single dimension to this decision".
"There are lot of factors... The prime minister is campaigning and of course there are a lot of issues in Tamil Nadu," Khurshid told NDTV.
"Many other important things that are on the prime minister's plate right now. So there could be a lot of reasons that went into this decision.
"I don't think we should look at any one single dimension to this decision. Let's not forget that the prime minister doesn't always go to Commonwealth (meetings)," he added.
The Commonwealth Heads of Government Meeting (CHOGM) of 53 countries is being held Nov 15-17 in Sri Lanka.
Political parties in Tamil Nadu, including both the AIADMK and DMK, were bitterly opposed to the prime minister's visit to Sri Lanka because of what they said were human rights abuses involving Tamil civilians.
CHOGM of 53 countries is being held Nov 15-17 in Sri Lanka.
Manmohan Singh Sunday sent a letter of regret to Sri Lankan President Mahinda Rajapaksa, explaining why he was not attending the summit.
Asked if domestic politics prevailed over India's diplomatic and strategic interests, Khurshid said: "We have a very important relationship with Sri Lanka. I don't think we should see this as a switch-on-switch-off affair."
He said the issue of attacks on Indian fishermen by the Sri Lanka Navy would be taken up with Colombo.
"Issues that are important to Tamil Nadu and to people in other parts of India also need to be pursued and addressed. The fishermen issue is important," he said.