Gilani ordered an operation by the police, paramilitary Pakistan Rangers and other law enforcement agencies while chairing a meeting of the Sindh Cabinet in Karachi.
The move coincided with unabated violence in Pakistan's largest city, with TV news channels reporting the death of at least 13 people.
"The time has come to take action in Karachi. If we don't take action now it will be too late," Gilani was quoted as saying by Dawn News channel.
The situation in Karachi can be tackled by the Pakistan Rangers and police and there is no need to call in the army, he said.
Gilani said the federal government was ready to help the Sindh government to restore peace "but the bleeding of Karachi must stop forthwith", an official statement said.
He directed the Sindh government and heads of law enforcement agencies to take "stern and indiscriminate action" against those behind the unrest in the city.
"This is unacceptable and the culprits must be brought to justice by awarding them exemplary punishment," Gilani said.
Sindh Home Minister Manzoor Hussain Wasan told the media that the first phase of the operation would focus on nine areas of Karachi, including Lyari, Korangi, Qasba Colony and Kati Pahari, where violence has escalated in recent days.