Rometty, 54, who is a senior vice president at IBM, will take over from current president and chief executive Samuel Palmisano, 60, on January 1, IBM said in a statement on Tuesday.
Palmisano, who was named IBM's chief executive in 2002, will stay on as chairman of the board of directors.
IBM said Rometty had also been elected to IBM's board of directors, effective immediately.
Rometty will be the ninth chief executive in the 100-year-old history of the company known as "Big Blue" and the first woman to hold the top post.
She joins a select group of women heading US technology companies. They include Ursula Burns at Xerox and Meg Whitman, the former chief executive of eBay who was recently named to the same position at Hewlett-Packard.
Carol Bartz was appointed the CEO of Yahoo! in 2009, but was fired on September 6.