A section of Congress leaders, including legislators, has demanded that the chief minister should accept responsibility for the defeat and resign.
But Virbhadra Singh, who is chief minister for a record sixth-time, said it was simply a "Modi wave" that demolished the Congress.
The Congress lost all four Lok Sabha seats to the Bharatiya Janata Party (BJP). The losers included the chief minister's wife Pratibha Singh.
Having been elected in 2009 from Mandi, Pratibha Singh was defeated by BJP's Ram Swaroop Sharma by 39,796 votes.
"It was no doubt a Modi wave. But the wave was strengthened by poor governance and lack of coordination between the government and the party," a senior Congress minister told IANS.
He said the BJP struggled hard to regain the ground lost in the 2012 assembly election by opening the doors for party rebels.
"On the other hand, the Congress kept dithering about its expelled leaders including five former legislators due to differences between Virbhadra Singh and state party president Sukhvinder Sukhu," said the minister who didn't wish to be identified.
A day after the May 7 polling in the state, the chief minister admitted that the pick-and-choose policy adopted vis-a-vis the rebels could go against the party.
"Had the party opened its doors to all the rebels who were keen to rejoin the party, the prospects of our nominees would have brightened," Virbhadra Singh had said.
Said a party leader: "It is simply overconfidence ... that went against the party. Power was confined to a few leaders."
Over the past two decades, voters in Himachal Pradesh have traditionally favoured the ruling party during general elections. It is for the first time that the electorate has acted differently.
The pro-BJP wave was so strong that the party led in 59 of the 68 assembly constituencies.
Even in the chief minister's Shimla (rural) constituency, the opposition got a lead of over 1,700 votes.
Congress insiders believe it is high time the government and the party review their policies and programmes.
"The Lok Sabha results are a referendum on the 15-month-old government. Had the government performed well, the Modi wave could have been countered to some extent," said a party insider.
The fissures within the party surfaced twice at vice president Rahul Gandhi's election rallies in the state May 1 when his "young brigade" lodged a complaint that they were deliberately sidelined.
At his rally in Bilaspur town, Gandhi took with him second-time legislator Rajesh Dharmani to the dais where the chief minister and other senior leaders were sitting.
Dharmani, despite being a local legislator, had not been invited to share the stage.
Similarly, a group of young 'turks' apprised Gandhi at his Solan rally that they were not getting due respect from party leaders in the state.
"Rahul got angry and assured us not to worry," one of them, Sandeep Kumar, told IANS.
The fallout of Dharmani's 'disregard' was that he resigned as a chief parliamentary secretary May 10, two days after the polling. He said he had quit as no work had been allocated to him.
Congress leader and former state general secretary Kuldip Rathore said the Lok Sabha result was a mandate against the government.
Virbhadra Singh has, however, ruled out his resignation.
In 2009, the BJP, which was in power in the state, won three of the four Lok Sabha seats.