Kanda, who is facing trial in the air-hostess Geetika Sharma suicide case, said that he would be floating his own party on Friday.
Today, the 46-year-old legislator met Governor Jagannath Pahadia at Raj Bhawan here and handed him his letter of withdrawing the support.
The move comes over a month after he got bail from a Delhi court in the suicide case. The legislator accused the chief minister of betraying him during the crises.
"When Congress government fell short of majority in 2009, I was the first to extend support to it," he said. Kanda, however, said that his "trust had been betrayed" and there had been occasions when Chief Minister Hooda did not stand by him.
Mentioning the ongoing case against him in the Delhi court, he said,"at the very outset Chief Minister Hooda should have asked for an inquiry of the suicide note of Geetika Sharma when it had surfaced as it involved one of his ministers, but he had not done so."
"The Chief Minister should have got the inquiry of this letter done," he said. Kanda was a Minister in the Hooda government before he was forced to step down after his involvement in the case in August 2012.
"I will not speak much about the case since the matter is sub-judice, I have full faith in the judiciary and the truth will prevail," he added.
After submitting the letter, Kanda addressed a press conference and said that he would be launching his political outfit-- Haryana Lokhit Party-- at Gurgaon on May 2.
Kanda, who has been Home Minister of the state earlier, also accused the Hooda government of tapping the phones of its Ministers and MLAs.
He claimed that Hooda's ministers "were powerless" and had no say in running of affairs of the government.
"As Home Minister (earlier), I knew that my phones and that of the ministers and some MLAs were being tapped and I brought this to the notice of the Chief Minister since it was a sensitive matter, but nothing happened," he said.
However, there is no threat to the Congress government in Haryana, which has 45 members that includes the five legislators of the Haryana Janhit Congress who joined it four years back.
The government also have the support of six other Independent MLAs and lone BSP member in the 90-member House. The Haryana Assembly polls are due in October.
After the resignation of Congress MLA from Sohna, Dharambir Singh, who quit last month to join the BJP and resignation of SAD's Charanjit Singh who quit his seat to join INLD, the effective strength of the Assembly has been reduced to 88 at present.
Kanda said he would like other MLAs supporting the government to take their own decision as he "would not like to be blamed for the fall of the government."
"Let other legislators decide for themselves. I would not like to be blamed by the government for its fall. The Hooda government which failed to carry out any development works during past ten years can take another three months if it wants to do any miracle, but it should not be given any chance to make excuses," he said.
Kanda claimed that his party would provide people a clean alternative which would be free from caste politics. "The people of Haryana are looking for a viable alternative. The state's politics has been under the control of two or three political families (Chautalas, Bhajan Lal, Bansi Lal and Hooda families), we want to change this," he said.
Kanda was instrumental in bringing the six Independent MLAs together to support the Hooda government when it had fallen short of majority after the 2009 Assembly polls.
To another question, Kanda said that he met Venod Sharma, who recently quit Congress, in Gurgaon, but refused to divulge what transpired between the two.
On floating the new party, he said "successive Chief Ministers have only nursed people from their own castes... Haryana belongs to all those who live in it. They have equal rights over its resources, the government they elect and jobs available.
My party will address these issues". He also said they had taken the feedback from the people over the last three months and then decided to launch the outfit.
"A new party comes into existence after people's trust has been betrayed. Our party will strive to build a new Haryana. We don't expect any overnight magic to happen, but yes we have seen when people's anger rises, it can shake governments and big parties like we recently saw in Delhi's case (where AAP sprang a surprise)," he said.
To another query, he said that his late father, Murli Dhar Goyal had been jailed for a brief period in 1948 after being charged with propagating the "RSS ideology".
He also said that though he may have supported the Hooda government, but he never had "Congress thinking". Congress legislator Venod Sharma was recently expelled by the party but he continues to remain the party MLA in the Assembly, sources said.