Transatlantic link typo by Sweden's Telia broke Cloudflare in the US

Re: More CloudFog

"The issue is related to a specific transit provider and we are working on temporarily disabling this provider to route around the issue,"

Sounds pretty redundant to me.

Private Networks

...and this, boys and girls, is why people still build private networks, even though they are expensive. Although it does depend on things like service providers having redundant route-reflectors in the core of their MPLS networks; and truly separate fibre paths between P-nodes. Don't ask me why these points are foremost in my mind.

Some large companies with deep pockets buy private networks from multiple suppliers to try and guarantee redundancy of equipment; but unless you buy pairs of separate fibre routes from each individual supplier, you can never guarantee that both providers haven't bought a fibre on the same path from a third-party supplier. Even then, good luck with getting four geographically separate entry points into each of your buildings. If you have an anonymous building in a random business park on the edge of a typical large town, you can usually only get, if you are lucky, two paths. There's no money in providing very high degrees of diversity, separacy and redundancy to everywhere. There's actually a surprisingly small number of highly-connected datacentres around.

Half hour ping.

No, no, the server wasn't down for 20 minutes, the Intertubes were slow.

A lot of Cloudflare services depend on actually being able to contact the site behind it, as opposed to hosting it directly.

If the Intertubes are broken, then it can't do so. This was the sort of error that really breaks normal redundancy as well - the prefixes were still in BGP but traffic wasn't delivered.

There isn't really any way to automatically protect 100% against that. Certainly not when some traffic via the transit provider is getting through and other traffic isn't - you really need a human to make the decision in that case. Especially since the only way to fix it is killing the entire transit connection, and doing so comes with its own risks (flap damping, for example).

Not just Cloudflare

A huge chunk of transatlantic traffic was affected. I understand it might be hard to grok for commentards but the Reg hacks have no excuse....