Speaking to reporters here, Yeddyurappa said he had dropped the idea on the advice of religious leaders and BJP President Nitin Gadkari.
Gadkari had advised Yeddyurappa that it was "not appropriate to use religious places, especially temples, for politics" and drop his temple face-off with Kumaraswamy on the issue of an alleged compromise bid by the Karnataka Chief Minister.
Yeddyurappa said he will visit the temple but not "swear" before the deity and just offer worship.
He said he would file a defamation case against Kumaraswamy for hurling fresh charges of irregularities to the effect that his family purchsed a cement manufacturing plant in Belgaum.
He said he would also send a legal notice to Congress leader Siddaramaiah for his charge that he and his family evaded income tax to the tune of Rs 360 crore.
Yeddyurappa said none of his family members would accompany him to Dharmasthala and some ministerial colleagues will be present during the pooja.
Yeddyurappa had challenged Kumaraswamy to come to the Lord Manjunatha temple at Dharamasthala along with his family and take the "trust test" to prove his claim that he (Yeddyurappa) had sent an emissary to "buy peace" with him. The programme has drawn flak from political and religious leaders.