In an overwhelming vote, Mattis became the first confirmed member of Trump's cabinet, hours after Trump was sworn in as the 45th US President, Xinhua news agency reported.
Earlier, Trump signed a waiver legislation after his inauguration to clear a legal barrier for Mattis to take office.
Under the US law, a former service member is barred from running the Pentagon less than seven years after retirement from military. Mattis only retired in 2013 as the Commander of the US Central Command. Trump signed the measure to grant Mattis an one-time exception.
The Senate Armed Services Committee voted 24-3 on January 12 to approve Mattis' nomination, moving the vote onto the Senate floor. The committee also easily passed a waiver exempting Mattis from the law that could prevent him from becoming the Pentagon chief.
At the hearing, Mattis took a clear anti-Russia stance during the hearing, describing Moscow as a "principle threat" to US security, a position notably different from Trump.
The United States should also check Russia's growing influence in the Arctic region, Mattis said.
Mattis also chose to distance from Trump on other key issues, including the role of the North Atlantic Treaty Organisation, which Trump said should shrink, and on the Iranian nuclear agreement, which Trump threatened to scrap.