According to reports, militants have entered the site containing prized ruins, reports say.
So far no reports of destruction of artefacts have been received, BBC reported on Thursday citing Syrian Observatory monitoring group.
The reports also said that IS also controled the nearby air base, prison and intelligence headquarter. Syria has admitted it has pulled pro-government troops out of the city.
Syrian state media said pro-government forces had been pulled out of Tadmur, the modern settlement on Palmyra - a Unesco World Heritage site - after "assuring the evacuation" of most of its inhabitants.
The Syrian Observatory for Human Rights also said that more than 100 pro-government troops were killed in overnight clashes around Palmyra.