The win left the Greeks as group runners-up ahead of the Russians on the head to head rule, both having four points to six for group winners the Czech Republic, who beat co-hosts Poland 1-0.
The Russians, 16 matches unbeaten, had most of the possession against a Greek side which had lost their last four straight meetings with their rivals and Alexander Kerzhakov and then Andrei Arshavin both sent early efforts off target against a side needing a win to have any hope of advancing.
Russia's top-scorer Alan Dzagoev met Andrei Arshavin's cross but headed it inches wide of the far post. Frustration mounted over the Russians as they stormed forward to get an equaliser, which would take them through to the knockout stage. But Despite having much of the ball in the second period and 26 goal attempts, the Russians couldn't bring in their first game form.
But Karagounis, a 35-year-old veteran of the Greeks' 2004 shock final win in Portugal, drove home right-footed to stun Dick Advocaat's side.
However, the Greeks were second best coming into this match after securing only 1 point in their first two games. But all they needed was a solitary goal from their inspiring 35-year-old skipper Giorgos Karagounis.
The Russians began their tournament with a 4-1 demolition of the Czech Republic in their tournament opener and then held co-hosts Poland to a draw but they failed to find their form in the second half as Karagounis looked to have won a penalty for the Greeks.
Tackled by Sergeo Ignashevich, the veteran went down but was then booked for diving - meaning he will miss the quarters. Giorgos Tzavellas then hit the bar as the Greeks tried to increase their advantage - but they held out for a famous success.