The Green party is set to form the first government under its leadership in the southern state of Baden Wuerttemberg after winning 24.1 per cent of the votes, more than double the votes polled in the last elections in 2006.
Chancellor Merkel's party suffered its worst defeat in the state election when it was voted out of office in Baden Wuerttemberg after nearly 60 years in power.
The CDU's debacle is seen by poll analysts as a rebuke by voters angry over the federal government's nuclear policy, especially its decision to extend the life span of the country's 17 nuclear reactors on an average by 12 years.
In the wake of the crisis at Japan's Fukushima reactor complex, crippled by a powerful earthquake and tsunami two weeks ago, Merkel's government had ordered a three-month suspension of its nuclear policy and temporarily shut down seven oldest reactors.
The CDU polled 39 per cent of the votes, around 5.2 per cent less than in the last election in 2006, but remained the largest party in Baden Wuerttemberg, according to provisional official results.
The Social Democratic Party (SPD) received 23.1 per cent of the votes, slightly less than in the last election five years ago. The Green Party and the SPD have a narrow majority to form a coalition government in Stuttgart, replacing the CDU-led government of premier Stefan Mappus.
The Green Party's leading candidate Winfried Kretschmann is almost certain to become the next premier.
In the state of Rhineland Palatinate, the SPD lost its absolute majority, but the state premier Kurt Beck will remain in office by forming a coalition with the Green Party, which polled 15.3 per cent votes, nearly quadrupling the votes it received five years ago.
The outcome of the elections cemented the Green Party's rise as the third major political force after the CDU and the SPD. The Free Democratic Party (FDP), which has been playing the role of "king-maker" in the German politics for several decades, has been pushed to the fourth place after losing a series of state elections.
The FDP lost 5.4 per cent votes in Baden Wuerttemberg, but managed to remain in the state parliament in Stuttgart by winning 5.3 per cent votes, slightly above the minimum five per cent needed. However, the FDP failed to cross the five per cent hurdle in Rhineland Palatinate.
Leaders of all political parties agreed that the disaster at the Fukushima reactor complex, caused by earthquake and tsunami influenced the outcome of the elections.
"The tragic events in Japan helped the Green party to mobilise voters against us," said Mappus, who was in office only for 14 months.
Sigmar Gabriel, Chairman of the SPD, said the outcome of the elections is a warning to Merkel's coalition government that its nuclear policy cannot be continued.