Everest outage was caused by split brains

Aww shoot

I only clicked on this article because I though it was about MUMSNET. I want my outrage back!

basic electrical mistake there: full load =/= start-up load

"full load" is almost never equal to "everything is starting up at once" load...

electrical load when a device starts up has a BIG load spike, sometimes this can even reach (for a few fractions of a second) 500% or even more of the regular load during normal operation

https://en.wikipedia.org/wiki/Inrush_current

Regular servers usually have a bios setting to delay automatic power on by a random amount of time.. usually this is decided when power is applied to the pre-power-on circuits and it's randomly chosen between 0 and 60/120 seconds - depending on the bios setting.

If you have just a few servers then this setting is very much recommended in order to spread the start-up load spikes over a longer period of time.

For a datacenter though, where they have a lot of servers starting up with random delays but all within the same period of 60 or 120 seconds from when the power is turned on, the combined load spikes from all those servers translates into a very LONG start-up load spike and the generators should be designed for a constant current load of at least 200...300% of the regular "full" load of the datacenter.

A lesson in redundancy

To have redundant everything you need to know what everything is. In the complex interconnected world in which we operate it's often hard to know what you don't know.

One of our servers at Memset (who are normally an all round top notch provider) was affected by this outage. Anticipating that data centres can never actually have 100% up time, we replicate our servers to another DC run by a different service provider. That way we can just switch the DNS and hey presto.

For our DNS we use another normally top notch provider with loads of DNS servers spread around the world etc.... It just so happened that after ten years of flawless and uninterrupted service they had a problem at the exact same time as the Memset outage. All DNS servers were running normally, but their control panel went offline for an hour due to a database glitch - meaning we couldn't switch the DNS to our redundant server.

As it happened, our Memset server came back very quickly and we didn't need to switch, but still, another lesson learned.

Any tips on how to mitigate against this problem would be much appreciated. DNS secondaries with another provider (or our own) would not have helped in this instance as DNS was running normally. We just couldn't modify the zone files.