The unilateral move by the Italian government follows the failure of direct negotiations with the Indian government aimed at resolving a case which has severely strained diplomatic relations between the two countries, the foreign ministry said in a statement.
India insists that the fate of the two marines, Massimiliano Latorre and Salvatore Girone, must be resolved by its courts because its citizens were shot in its territorial waters.
Italy maintains that the shooting, which happened while the two marines were serving as part of an anti-piracy mission, occurred in international waters off southern India and should be dealt with under the UN Convention on the Law of the Sea (UNCLOS).
Under UNCLOS, parties to a dispute can seek arbitration through an ad hoc panel or the creation of a special tribunal. Other alternatives are to seek a judgement from the International Court of Justice in the Hague or the International Tribunal for the Law of the Sea, in Hamburg, Germany.
Italy's foreign ministry did not specify the exact procedure it was starting but said they would be making an immediate request for both the marines to be allowed to stay on home soil pending the outcome of the arbitration procedure.
Latorre has been in Italy since last year having been granted permission to leave India on health grounds due to a heart condition which required minor surgery.
Girone remains confined to Italy's embassy in Delhi pending a trial which Italian officials say has taken far too long to come to court.
The marines have denied shooting the fishermen deliberately but have conceded they mistook the fishing boat for a pirate vessel and fired what were supposed to be warning shots.