The boat capsized off the Ahwar district in Yemen's southern Abyan province on the Arabian Sea, security sources said.
Abdul Qadir Bajameel, a provincial health official, said 10 of the around 150 people on board were rescued - nine Ethiopians and one Yemeni - but dozens remained missing. Two medics said rescuers were still looking for survivors.
The International Organization for Migration says Yemen continues to witness a significant increase in the influx of irregular migrants arriving from Africa.
Migrants cross the Bab al-Mandab strait that separates Djibouti and Eritrea from Yemen each year on flimsy boats in the hope of reaching Saudi Arabia and other Gulf countries in the hope of finding work.
The IOM describes the route from the Horn of Africa to Yemen as "one of the world's busiest and most perilous mixed migration routes". It said it recorded the arrival of more than 60,000 migrants in Yemen last year.
(Reporting by Mohammed Ghobari and Reyam Mokhashef; Writing by Hatem Maher; Editing by Alison Williams)

epa01146798 (FILE) A file photograph dated 08 October 2007 of Haziz, an economic migrant from Ehtiopia's Ogodan region, standing next to a boat similar to the ones that he has previously taken to cross to Yemen from Bosasso, Somalia. He has made this dangerous journey five times, but each time got sent back by the Yemeni authorities. As many as 2600 migrants from Ethiopia and Somalia will pay $80 each to make this dangerous journey - hundreds die along the way and even when they reach the Yemen there is no guarantee they will find work or enough money to take them on to Saudia or Dubai, their final destinations. EPA/KATE HOLT