The Counter terrorism Unit-Japan will be led from the prime minister's office but will operate within the foreign ministry. Its creation was originally planned for next April but was brought forward in the wake of the November 13 massacre in Paris, the Japan Times reported.
The government aims to strengthen counter-terror capabilities and to step up cooperation with the intelligence organs of other countries.
"The situation in international terrorism has never been more severe," Prime Minister Shinzo Abe told a ministerial meeting earlier in the day.
"With the (G-7) summit and the Olympics and Paralympics in sight, our country must take the best possible measures in cooperation with the international community," Abe said.
The staff of 20 will be drawn from the foreign and defence ministries, the National Police Agency and other bodies.
Separately, around 20 terrorism experts and regional affairs officials will be deployed to diplomatic missions overseas.
The missions will be in four regions: the Middle East, North and West Africa, Southeast Asia and South Asia.