Rackspace flattened by Texas trucker

Resilience

I thought resilience meant "what if the data centre was nuked?" as well. Surely this is a single point of failure. Given what I have seen and heard of American driving standards perhaps this is not a perfect storm. I understand the phobia Americans have for small cars is widely based on a fear of being crushed by amphetamine-crazed long distance waggon drivers.

System Uptime

I work in broadcast and my working ethic is "if the listener/viewer can tell that somthing is wrong i have failed in my task"

The studio can be on fire but as long as we're still transmitting a programme then its OK. Broadcast systems CANNOT ever have downtime, in this instance it's not RackSpace's fault, but power cuts are a problem and obiously although their backup generators kicked in, the systems did not start up correctly, which im sure they will fix.

Syncing sites is an expensive (but good) way to go, if they're to compete in the hosting market then they have to tradeoff the cost of the second redundancy against how much they can charge.

Rackspace's fault

@Dave

No, sorry, if the servers go down, it's Rackspace's fault. Period. It's their job to make sure the servers stay running and they failed.

A total power failure is to be expected at some point and should (and can) be covered, and wasn't.

Too much power, not a lack of it

The problem wasn't lack of power generation capacity. It was that the power had to be removed to allow for recovery and repair work to be done safely. You might like uptime but electrical workers like their lives.

All of this reinforces what ISPs have known for years: site redundancy is the best redundancy of all.

@ Glen Turner

"It was that the power had to be removed to allow for recovery and repair work to be done safely. You might like uptime but electrical workers like their lives."

Sorry, but I just can't see the relevance of that statement.

The safety of the emergency workers should have nothing to do with whether or not a service provider like Rackspace has AC power available or not - if you're in that line of business you should have all of the following - duplicated utility feeds from different distribution sub-circuits, at least triple redundant backup generators (where I work, of the 3, one is on load, one is running at speed but not on load and the other is waiting to start should either of the first 2 fail during a utility failure) and significant UPS capacity that can run all critical systems, including aircon, for long enough to get a failed engine sorted should all three not start.

Being in the telephone exchange business, we have all of these plus several hours of 50 volt battery capacity available.

Is this cheap? - of course not - if nothing else the building I'm based in consumes about 1000 litres of diesel/hour when the power is out, but if you're in the reliability business need to spend the money.

Oh, and yes, there have been occassions when we were down to battery power and UPS only, ie., all 3 engines failed in turn during a power outage.

...its BAU P&L...

@Cube: 2 independent data centres

... yup, better than one... it sounds like they do not have a

sync'd replication scheme to another site, since they were hell-bent on getting the power up on the affected DC

@Graham: 100% network uptime guaranteed

...looks like the marketing department got confused with all

the 99.9995 stuff and thought "thats close enough to 100.. and its less numbers so the advert won't cost as much"

Catastrophic events will occur, it just depends on what you are prepared to invest in as a company to ensure you *still* have a client base after a major outage... fully agree with @Dave: its a financial tradeoff

BBC (and Dave can probably verify)

A mate who used to work at the Beeb told me that an IBM salesdroid was horrified that the BBC wouldn't buy UPSs for their new servers.

He was taken to a basement where he was shown an engine, from a ship, which powers the whole of Television Centre if mains power goes out. The 'starter motor' for this giant generator/engine is apparently a V8 car engine...

The best-laid plans

Well, once again - thanks to all the great minds that contribute here - Rackspace has been duly castigated for its lack of foresight/money/intelligence/honesty/divination.

However, back in the real world - where business continuity planning rules apply - we learn two things: We learn that the amount we're prepared to pay for redundancy, alternate power sources, etc. bears a direct relationship to our estimation of the likelihood of an event occuring (and we are often made fools of by doing so) and; We learn that Robert Burns may well have been in the BCP business. Why else would he have written the cardinal rule of business continuity planning, "The best laid plans o' mice and men gang aft agley."?

By the way, congratulations to Chris Collins for the day's most gratuitous swipe at America and Americans. Way to go, Chris!

Strategy for prevention needed - Rackspace could up the game.

You've all mentioned how Rackspace needs to secure it's backup electrical facilities, but this incident highlights their need to follow the less traveled path of investing in the improvement of their primary commodity.

In much the same way that Rackspace might build a bleeding edge facility, the company (and it's like) should blaze the path for alternative energy sources. I don't mean backup alone, but as a primary power source, using bleeding edge power generation technology.

Find another company that is on this patj, invest in them, and reap the rewards for using their breakthrough method. Renewable energy issues aside, the purpose is to further reduce dependencies.

Sure, they likely receive kickbacks from their home town headquarters for using the local power stations and contributing to town treasury. But not-spending is the best way to save money - not paying discounts - and that means higher revenues (further enhancements, etc).

The larger benefit (the one I'm most interested in) is that of the alternative energy industry. When big companies risk-and-research (invest in burgeoning industries) it can quickly trickle down to city governments and individual households.

If I wanted to reassure customers, I'd announce investments/partnerships with an alternate energy outfit, pointing to a goal to be a umpteen-percent less dependent on local power.

The best competitive advantage is when you're the only player. Rackspace should see this opportunity to do more than fix it, and prevent it.