Manjhi along with 17 ministers took oath of office at a simple ceremony at Raj Bhavan, officials said.
Manjhi was Monday named successor by outgoing chief minister Nitish Kumar after he was authorised by the Janata Dal-United legislature party to elect a new leader. Nitish Kumar refused to withdraw his resignation after his party's debacle in the Lok Sabha polls.
Manjhi, a graduate, began his life as a shepherd to earn his livelihood. His father Ramjeet Ram Manjhi was a landless labourer.
"It is a fact that Manjhi in his childhood worked as a shepherd for a farmer in his native village," a close relative of Manjhi told IANS here.
Manjhi along with Nitish Kumar and JD-U chief Sharad Yadav Monday met Governor D.Y. Patil and staked claim to form the government.
Party leaders said Nitish Kumar chose the soft-spoken Manjhi as he is a non-controversial leader with a clean image.
Manjhi - from the Mushahar community - was SC/ST welfare minister in the Nitish Kumar cabinet. He is legislator from the Makhdoompur assembly constituency.
"For the first time, a leader from the Mushahar community, poorest and at the bottom of the socio-economic category, has become chief minister," JD-U spokesperson Nawal Kumar Sharma said.
Manjhi also contested the Lok Sabha poll from Gaya constituency but lost.
A native of Mahkar village in Gaya district, Manjhi is the third Dalit to become chief minister after Bhola Paswan Shastri and Ram Sundar Das.
The JD-U earlier said it has the majority to form the government with support of the Communist Party of India and Independent legislators.