Sant Vivek Das Acharya, the chief of Kabirchaura Mutt Trust, also severely criticised Modi for trying to "paint" Varanasi as a centre of Hindu religion and said the current political discourse may have a long-term impact on the city's "soul and identity".
"It is very unfortunate for people of Varanasi (that Modi is contesting from here). He is divisive. Modi does not reflect the composite culture of the sacred town," he told PTI.
Kabir Chaura Matt is considered the most sacred place by Kabir Panthis as devotees from across the globe come here, the birthplace of the 15th Century saint -Sant Kabir.
Significantly, two key Hindu religious leaders -- Puri Shankaracharya Swami Adhokshjanand Devtirath and Dwarka Shankaracharya Swami Swaroopanand Saraswati -- have also came out against Modi contesting from Varanasi.
Modi is pitted against AAP leader Arvind Kejriwal and Congress candidate Ajay Rai, among others, in the temple town where election will be held on May 12.
Terming Varanasi a confluence of various religious beliefs and symbol of India's core values, Acharya accused Modi and BJP of trying to exploit electoral gain by taking a "narrow view" of the city that it was a nerve centre of Hindu tradition.
"It is a nerve centre of various religions, traditions and ideologies. Kabir had made the city a centre of Indian civilisation. Buddha had come here 2,500 years ago. It is an important place for people belonging to Jain faith," Acharya said.
"They (BJP) will disturb the Hindu-Muslim unity. An artificial wave for him (Modi) is being created. I will not explain, but it is well known how it is being done. How people are being bought over," Acharya said.
The religious leader said if people of Varanasi have not understood the "misinterpretation" of the city's values, then it will be unfortunate for the whole city. "The place reflects India's core values of pluralism, tolerance and intellectual thinking.
The current political discourse may have a long-term impact on the city. If people here have not understood the threat, then it is a misfortune for the city," he said.
Acharya said the incumbent MP Murli Manohar Joshi did not do any work for the city during the last five years.
"If one of BJP's tallest leaders did nothing for city, then how can we expect Modi to do something."
The Mutt chief also justified his political observation saying, "We are saints, but at the same time we cannot remain oblivious to important issues facing the country. We are not inclined towards or associated with any political party."
The devotees at the Mutt also felt that the the "peace" and tranquility in the holy city may be affected due to the high-pitched electoral battle.
"Varanasi has a unique identity in the whole world. Everybody must try to protect that. The politicians should not be allowed to exploit the city for their benefit," said a devotee while offering prayers at Kabir-Hut, from where the saint used to talk to his followers.
The Mutt is involved in various activities and projects to propagate the teachings of Kabir in various parts of the world. BJP decided to field Modi from this temple town so that it helps the party in politically crucial Uttar Pradesh and Bihar.
Kejriwal had announced he would contest from Varanasi after BJP decided to field Modi. The AAP leader, who had led his party to a spectacular victory in the Delhi Assembly polls in December last year, has been trying to replicate the "successful Delhi campaign model" here by holding scores of roadshows and corner meetings across Varanasi.
Hundreds of AAP volunteers are also making door-to-door visits to garner support from people in this holy city where BJP has been running an "aggressive campaign".
There have been a number of clashes between AAP and BJP workers in the city. Last week, AAP leader and former Delhi Minister Somnath Bharti was assaulted allegedly by a group of BJP supporters.
In 2009 Lok Sabha elections, senior BJP leader Murli Manohar Joshi had won the seat after defeating Mukhtar Ansari, who is currently in jail in a murder case, by a margin of 17,000 votes.
Quami Ekta Dal's Ansari, known to have a strong support base among Muslims, has decided not to contest against Modi from the seat "to avoid division of secular votes".
Also in the fray is Samajwadi Party's Kailash Chaurasiya, who is the sitting MLA from nearby Mirzapur.