After stepping off a plane from the Brazilian host city, Tokyo governor Yuriko Koike carried the flag at a ceremony at the city's Haneda airport.
"I strongly feel a heavy responsiblity" for the next Olympics, Koike told the crowd. "I'm very happy that we're able to bring the flag back after more than 50 years."
Tokyo last hosted the summer Olympics in 1964, underscoring Japan's post-war coming out party as it grew into a global economic powerhouse.
A kimono-clad Koike on Sunday received the flag at the closing ceremony in Rio where thousands of fans and athletes donned ponchos on a wet and windy night for a colourful festival of Brazilian culture and music with bursts of fireworks.
Japan's Prime Minister Shinzo Abe made a comical cameo as Nintendo video game character Super Mario as Tokyo set a light-hearted tone for its hosting of the Games in four years. Abe came out from a pipe after a video showed plumber Mario drilling down from Tokyo into the earth to reach Brazil.
"I wanted to show Japan's soft power to the world with the help of Japanese characters," Abe told reporters.
"I wasn't sure how the audience would react. But I received so many cheers. I appreciate it."
Abe pledged in Rio that he will work hard to host the best Games yet but Tokyo's Olympic preparations have suffered high-profile setbacks including soaring costs and having to redesign the Games logo after accusations of plagiarism.
French prosecutors have also launched an investigation into alleged bribes linked to Tokyo's winning Olympic bid, which organisers have denied.
Koike, who was elected in July as Tokyo's first female governor, has ordered officials to rein in ballooning costs and pledged a formal review, as concerns grow over soaring costs which could potentially double or even triple from the reported original forecast of $7.14 billion.
The Games were awarded to Tokyo in 2013, with expectations that they would be a model of efficiency with the city touting itself as "peaceful, reliable, safe, and stable".
Tokyo's metropolitan conurbation is the world's largest with more than 35 million people, but streets are safe, trains run on time and the air is clean. And with strict gun control and a public honesty visitors find disarming, few people ever experience serious crime.
The games in Brazil -- which is embroiled in a political crisis over the impeachment of suspended president Dilma Rousseff -- suffered its own series of setbacks.
Tourists, officials and athletes have had to dodge the scenic city's notorious street crime, structural problems inside the Olympic Village were a challenge and the Olympic diving pool turned green.