Rao, who is on a visit to Delhi, Monday submitted his resignation to the Lok Sabha secretary general.
KCR, as Rao is popularly known, was elected from both the Medak Lok Sabha and Gajwel assembly constituencies.
As the TRS won a majority in the Telangana assembly, KCR has decided to retain the assembly seat. He was already elected leader of TRS legislature party.
The TRS leader will take oath as Telangana's first chief minister after the formation of a separate state June 2.
The TRS bagged 63 seats in 119-member Telangana assembly. It also won 11 out of 17 Lok Sabha seats in the region.
KCR won from Medak by a margin of 3.97 lakh votes, the highest for any candidate in undivided Andhra Pradesh.
Meanwhile, Bharatiya Janata Party's Telangana unit chief G. Kishan Reddy is said to be keen to contest the by-election for Medak Lok Sabha seat. He was elected to the assembly from Amberpet in Hyderabad.
The BJP is likely to take a decision in this regard after consultating its ally the Telugu Desam Party (TDP).
Under the seat sharing agreement between the two parties, the BJP had contested Medak seat but its candidate C. Narendranath had finished third.