ACP K Venkatesh Warlu told PTI that Rao's family identified his body at the government-run Gandhi Hospital on Friday after the local police alerted them.
Rao, who was 86 years old, was found in Begumpet area on June 24. He suffered from Alzheimer's disease and presumably could not tell the police about himself.
Police admitted him in Gandhi Hospital, where he breathed his last night while undergoing treatment, his identity not known to the doctors till the very end, the ACP said.
On May 29, Rao had gone missing from the premises of Saibaba temple in Shirdi, Maharashtra, which he was visiting with his family.
According to the noted ghazal singer Talat Aziz, Rao was one of the last court singers of the Nizam of Hyderabad, and he specialised in singing the Sufi lyrics by Amir Khusro.
Telangana government had invited him to sing at the function to mark the foundation day of the state on June 2, but Rao was missing then.
Rao's students had created a page on the Facebook titled 'Finding Pandit Vithal Rao'.