If the Philippine Islamist group is confirmed to be behind the kidnapping, it would be their second such hostage-taking in as many weeks and comes amid an uptick in such attacks.
Sabah police commissioner Abdul Rashid Harun told Malaysia's official news agency Bernama that authorities were still investigating if the kidnapping yesterday occurred in Malaysian or international waters.
"The area is vast and we have our assets there. So we are investigating whether it happened in or off our waters," he was quoted as saying by Bernama.
"The five other individuals in the ship comprising Myanmar and Indonesian nationals were released." There were conflicting reports over whether the boat, the size of which was not immediately known, had been sailing to Malaysia from the Philippines or making the reverse journey.
Many Western and other embassies routinely issue warnings against travelling to most of the Philippines' Muslim-populated southern regions, which lie just northeast of Sabah, because of the risk of being abducted by the Abu Sayyaf.
On Tuesday, officials said that 10 Indonesian sailors had been kidnapped in waters off the southern Philippines. The incident was believed to have happened last Saturday.
The Philippine military said initial information indicated the sailors may have been taken by an Abu Sayyaf faction to Sulu, a remote southern island that is a hideout of the militant outfit.
Someone claiming to be from Abu Sayyaf called the vessel's owners to demand a ransom for the sailors' release. Other recent kidnappings, including of two Canadian tourists and a Norwegian resort owner in areas previously considered beyond the group's reach, have further raised fears.
Last year, Malaysian Bernard Then was kidnapped from a seaside restaurant in Sabah, about 300 kilometres (180 miles) from the Abu Sayyaf's Jolo stronghold. He was later killed by the group.