Japan's Foreign Minister Seiji Maehara and Indian Commerce Minister Anand Sharma signed the deal in Tokyo, hoping it will boost two-way trade which totalled 900 billion yen (USD 10.7 billion) in 2009 - less than one per cent of Japan's total foreign trade.
"We have no doubt in our minds that this will usher in a new era of economic engagement, which will bring development, innovation and also prosperity in both societies," said the Indian commerce minister.
The trade and investment agreement, approved in-principle by both countries' leaders last year, aims to open new markets for Japan as its population ages and shrinks, and to fuel the rapid growth of emerging power India.
It will help Japanese auto-makers such as Suzuki by lifting tariffs on car parts shipped to its factories in India and ease access for Indian generic drug-makers to a lucrative market in fast-greying Japan.
India - which has already signed a free-trade deal with South Korea, Japan's export rival in autos and electronics, but not with China -will become Japan's 12th free trade partner.
The agreement, for which Japan hopes to gain legislative approval in the Diet by the summer, will immediately reduce Japan's tariffs to zero on almost all industrial products imported from India.
Tokyo also plans to scrap duties on some foodstuffs - including curry ingredients, pepper and tea -- within 10 years, but will maintain a high tariff wall to protect its politically sensitive rice sector.
India will cut trade barriers on auto parts gradually, as well as on Japanese steel, electronics and machinery products, eventually to zero.
But the South Asian giant, with booming auto sales to its growing middle class, will maintain tariffs on assembled vehicles.
A Japanese government official said: "India obviously wants to protect its own auto industry."
India will also ease access to Japanese single-brand companies, allowing them controlling stakes of 51 per cent in local entities, and giving them the right to set up franchises in India.
But in other sectors, the two countries only agreed to continue talks.
Japan, which tightly controls immigration, has so far failed to meet India''s wish to send nurses and caregivers to Japan, where almost one in four people is aged over 65 and the aged-care sector is suffering labour shortages.