That'll teach us to use cheap offshore hosting. A major undersea cable link between SA and the EU is having trouble, leaving internet service providers scrambling and customers fuming. Including yours truly.
For many years it was the saviour of South African internet users. The Sat-3 undersea fibre optic cable, properly known as SAT-3/WASC/SAFE, runs all the way up the coast of Africa to Spain. Before the arrival of johnny-come-latelies like Seacom, the vast bulk of internet traffic was moved either that way or via expensive, high-latency satellite connections. Even today Sat-3 still carries the majority of traffic between South Africa and Western Europe. And as of Wednesday it started having serious trouble – leaving much of that traffic homeless. Then, as traffic started being diverted to alternative links, those routes became saturated, and the knock-on rapidly reduced international access speeds for much of the country.
Worst affected were those with essential servers in data centres in, say, Germany. Companies like, say, The Daily Maverick (though it’s just our primary mail servers; luckily our web servers are staunchly American). And the many, many internet service providers that rely on the European link as a backup for more direct routes to the United States, through which still flows the vast majority of internet traffic.
Let’s put it this way. If one of the other undersea cables were broken by a ship dragging its anchor at the same time, or sudden sunspot activity causes trouble with satellite uplinks, then everybody would have been in deep trouble.
Telkom, the local operator of Sat-3 and the lead in the consortium that owns it, says this is a case of planned maintenance that went awry. Technicians were fiddling with power units in preparation for some work due to be done on the part of the cable between Accra and Abidjan – over the weekend, when the impact would normally be minimal. Something went badly wrong, leading to the outage. By late Thursday afternoon Telkom was estimating a return to full service within hours rather than days, but that doesn’t come with a guarantee.
Don’t even ask what would have happened if we were in the middle of the World Cup, with hungry Europeans soaking up high-definition (and bandwith-intensive) live video. True patriots don’t ask those kind of questions. Instead they just pray really, really hard.
UPDATE: We hear that several major financial institutions had complete international connectivity outages and that both Sat-3 and Safe, the portion of the cable network that runs past Madagascar to India and points east, went entirely dark for around one hour on Thursday afternoon. Which isn’t supposed to happen, because Safe is the redundant path for Sat-3, and some Telkom customers pay good money for that redundancy and the assurance that they’ll never lose all internet access.
However, we also have confirmation that links on Sat-3 are starting to reach normal capacity again, as of 19:00 on Thursday, so it looks like Friday may be a better day. At least for everybody who doesn’t have to explain to very important, very irate customers what, exactly, went wrong.
By Phillip de Wet
"The soul is known by its acts" ~ Thomas Aquinas