The 61-year old Sharma would be administered oath of office and secrecy by President Pranab Mukherjee tomorrow. Sharma a 1976-batch Bihar cadre IAS officer is currently the Defence Secretary.
Rai, who earlier drew flak from government and ruling parties for holding press conferences and interacting with media, today thanked media persons just before demitting office today.
"It is because of all of you that this institution has been able to function independently. I hope you continue to extend support to the institution," he said, but did not take any questions.
Sharma holds a Masters degree in Political Science from the University of York.
Like Rai, Sharma too has served as Secretary in the Department of Financial Services. He has spent over 10 years in the Defence Ministry in various capacities.
The CAG is appointed for a term of six years or till the incumbent attains the age of 65 years, whichever is earlier.
Sharma's appointment comes at a time when the CAG has come under sharp attack from the government for its assessment of Rs 1.76 lakh crore loss in allocation of 2G spectrum.
Under Rai's tenure, the CAG was accused of going beyond its auditing ambit and indulging in analysing policy decisions.
Rai, however had hit back saying the government just wanted the CAG to be an accountant.
Just a few days before Rai's retirement, Information and Broadcasting Minister Manish Tewari said the chief auditor had done a "disservice" to India story by bringing out "hugely exaggerated and mythical numbers" in public discourse.
The new CAG, Sharma, will have to handle the audit report on AgustaWestland chopper deal, among others. He was closely associated with the deal from its early days till its completion in 2010.