It may not be as wild a guess as it seems at first glance. For of late "the bear is loose" - to use Obama's own catch phrase - for his recent impromptu escapades from the "bubble" at the White House.
Only last Tuesday, as news reports would have it, Chief of Staff Denis McDonough suggested an afternoon coffee break, and without telling the press or most of the staff, the president walked a block and half to a nearby Starbucks outlet.
"The bear is loose again," Obama told a staffer as the two walked out without jackets and with their shirt sleeves rolled up - tailed by White House reporters and the secret service in tow.
"How's the coffee?" a CBS News correspondent who just happened to be around, asked Obama as the commander-in-chief walked back to the West Wing with a paper cup in hand, greeting construction workers and waving at tourists on the way.
"Uhh...it's tea," the president replied.
The president's staff, according to CBS News, says there's no planned message in these impromptu outings, that he just likes to walk and talk, but they like the message they think it sends: regular guy.
Then next day Obama took Education Secretary Arne Duncan for a short ride across the Potomac to grab a hamburger for lunch at the FireFlies, a small single storey restaurant in Alexandria, Virginia.
Three weeks earlier too while heading over to the nearby Department of the Interior for a meeting, Obama "decided to break with tradition: He walked over instead."
"On the way there, he got to chat with all sorts of people -- from roadside vendors to tourists to locals. Needless to say, they weren't expecting it," as the White House blog recorded.
The President also cracked the same joke during that outing shouting to tourists that "the bear is loose."
"I don't get a chance to take walks very often," he then said. "Secret Service gets a little stressed."
"But every once in a while I'm able to sneak off. I'm sort of like the circus bear that kind of breaks the chain."
Obama may be currently at odds with the Russian President Vladimir Putin, but four years ago he took his predecessor Dmitri Medvedev for a ride across the Potomac for lunch at his favourite hamburger joint, Ray's Hell Burger, in Arlington, Virginia.
It was "Just a Couple of Guys Grabbing Burgers" as the New York Times headlined the summit lunch.
So would it be a surprise if Obama takes Modi out for a little tete-a-tete with a once unwelcome guest to let bygones be bygones?