Speaking at a public meeting in Junagadh on Tuesday, Mr Modi alleged that the Governor's residence had become 'Congress House'.
"It is very unfortunate that people whose only aim is to defame me are seriously damaging the public life in the state, and when Raj Bhavan becomes a hub of political conspiracies, it becomes the duty of the CM to tell the truth," Mr Modi said.
On Monday, Gujarat Congress president Arjun Modhvadia had attacked the government for its plans to host "Rs 800 per plate luncheon" on Independence Day despite 152 tehsils of the state being scarcity-hit.
In reaction to the remark, Mr Modi said, "We are committed to honour the post of Governor and Raj Bhavan, but here Raj Bhavan has become Congress Bhavan and conspiracies to defame Gujarat government are being hatched here."
"Actually, on behalf of the Governor, there is a gathering called 'At Home' in Junagadh today, and for that the Governor's office had invited tenders, and one tender was submitted for Rs 700 per dish of snacks and tea," he added.
Mr Modi said that it was the Governor's office which invited tenders and finalised the menu, while the state government only paid the bills.
Continuing on his tirade against Congress opposition in the state, Mr Modi said the party was promising "fraudulent schemes" to gather support from the people while referring to Congress' promise to provide affordable houses to women.
"Some fraudulent people are roaming around... offering houses to women of Gujarat. But why they aren't building houses for women in Congress-ruled Delhi?"
The chief minister also said that if state Congress had courage, it should distribute letters with the seal of the Union government or of the Prime Minister. "Otherwise this is a fraud scheme."
Targeting the Governor, Mr Modi said his government introduced and got passed a bill providing 50 per cent reservations to women in local self-governance bodies, "but the Governor, herself a woman, has declined to give assent".