The Uzbekistani wild card ended Djokovic's 15-match winning streak here with a 7-6(8), 5-7, 2-6, 7-6(5), 6-4 victory over almost five hours.
He will next play No. 30 seed Pablo Carreno Busta, who cruised past UK's Kyle Edmund 6-2, 6-4, 6-2 in the third round.
Djokovic, who saved six break points in a 16-minute opening game, saved one set point at 5-6 in the first set tie-break before failing to convert two of his own chances at 7-6 and 8-7.
The second seed regained control, but dropped to a 1-4 deficit in the fourth set only to stage a comeback.
Istomin, leading 5-4, held one set point opportunity on his Serbian opponent's serve at 30-40 and later took a 5-1 lead in the tie-break, prior to requiring a further three set points to take the pair's sixth meeting to a deciding set.
Istomin broke decisively to 30 for a 3-2 lead in the decider and the world number 117 held his nerve to complete a memorable win in four hours and 48 minutes.
Djokovic committed 72 un-forced errors as he saw his perfect start to 2017 come to an end.
"All the credit to Denis for playing amazing. He deserved to win. No doubt, he was a better player in the clutch moments. He stepped it up, played aggressive. Served very well, very precise," Djokovic was quoted as saying by the Association of Tennis Professionals (ATP) after the match.
"There's not much I could do. Of course, I was not pleased with my performance overall. But I have to congratulate my opponent today," he added.