Fire at The Planet takes down thousands of websites
A fire at The Planet's H1 data center in Houston, Texas on Saturday has taken out thousands of websites. In messages posted on the web hosting firm's forum, the company blamed a faulty transformer for the fire. No servers or networking equipment were damaged, but the data centre remains without power, after The Planet shut down …
Even the plane't own site seems to be down now as I can't access the status page link from this article either.
Try getting to http://forums.theplanet.com/index.php?&showtopic=90185&st=20
It's slow, but I bet there are thousands waiting for news.
I have around 25 sites hosted in the Houston centre, I'm surprised the phone hasn't started ringing yet !
To keep you up-to-date, here is the latest information about the outage in our H1 data center.
We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.
Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation.
We will update you again around 2:30 p.m. this afternoon.
###################################
2:30pm CDT is about 8:30pm BST, so an hour and a half till the next update.
They also have said :
"We absolutely intend to live up to our SLA agreements, and we will proactively credit accounts once we understand full outage times. Right now, getting customers back online is the most critical."
I once had a tour of a data centre in the UK and was shocked to find their fire extingusher system was "water sprinklers", not powder or gas. Apparently the insurance company requested that system. You don't see that system on the star ship Enterprise.
People should spread their servers around colocation sites - Use a provider in Edinburgh as well as London for example.
So much for a superior network when you cannot keep the power on!
Need I say SOFTLAYER at all! The Planet is run by muppets anyway
And not one of them has a recent backup to load Houston's data so the public can be served?
Phorm's server(s) is(are) there, are they not? :-)
Hmm , crap power supplier not adequately monitoring the grid delivery and sounds to me more likely the system mains power transformer has been running at 115% plus power overload margin for far too long !
Mind you ,I have seen and heard of at lest three go with a very spectacular bang in my area causing massive local power outages for hours on end !
We have a server there: http://www.cardesignnews.com
Funny we also haven't heard from any users yet... but maybe the nameservers being down will affect (outsourced) mail too? (Our servers are up and running in H2 but our nameservers are in H1).
oh dear, BT's phorm hosted / controlled server for the webwise system seems to have been a casualty as well, not a very resilient system BT
peter
My backup server in H1 is down. I noticed that it timed out last night when transferring from my primary servers. Damn lucky that all I use it for is backup and secondary DNS. I had a server down with 1&s****house1 last week I toyed with the idea of moving those server customers onto the backup server briefly before setting up a new server elsewhere. Glad I bit the bullet and set up a new server straight away instead. Even more lucky is the fact that my experience to date with The Planet has been so good that I nearly set up the new server with them.
I'm willing to bet that The Planet will get the entire Houston data center back on-line faster than 1&1 can get my server with them back on-line. But then thats not much of a bet the 1&1 server has been off-line for 14 days now! Muppets.
@Alan
It's the weekend. Come 10:32am Monday and if you're still off-line you'll know about it, that's how long it takes customers to realise that it's not their exchange server that's the problem.
I'm not a webhosting or datacentre guy, but I was under the impression that there would be procedures in place to guard against this sort of outage - offsite stuff, redundant sites, etc?
If that's the case, is redundancy what is going to happen to the DR guys? :-)
Alan - maybe no-one likes your websites and doesn't care? ;-)
Steven "Only joking Alan" Raith
it's nearly monday morning and b3ta isn't working
people may be forced to do some work if this isn't fixed soon
http://news.bbc.co.uk/1/hi/world/americas/7424571.stm
According to http://www.thehostingnews.com/news-dedicated-server-firm-the-planet-data-center-manager-garners-award-4306.html it claims "Mr. Lowenberg conducted a six-month trial to reduce power consumption and increase data center operating efficiency. Initial results demonstrate that while critical server loads increased by 5 percent, power used for cooling decreased by 31 percent. Overall, the company experienced power reductions of up to 13.5 percent through a broad range of improvements. The new green initiatives were conducted across its six world-class data centers." and also The Planet operates more than 150 30-ton computer room air conditioning (CRAC) units across its six data centers. In one data center alone, the company was able to turn off four of the units. The cooling requirement on two of the units was reduced to 50 percent of capacity, while another nine now operate at 25 percent of capacity. The company also extended the return air plenums on all of its down-flow CRAC units to optimize efficiency. "
So if the Phorm/Webwise system was operational, does that mean BT Broadband system goes t*ts up?
Oh dear BT, this Phorm/Webwise system is going to do wonders for your customer satisfaction. ... NOT!
BT customers could be leaving in Droves!
(They possibly will anyhow once they get a handle on the invasive nature of Phorm/Webwise interception of all their HTTP traffic and find out the history of Phorm (121Media) and its nasty spyware products.
Paris, Because she loves a warm fire in her belly and she frequently goes t*ts up.
We host a site of medium-high importance. It has been our plan that as soon as we can financially afford to set up a duplicate rack in a totally different location we will. Both sites will be load balanced and data replicated in real time. It adds at least 150% to the cost but if you need that level of resiliance you have to pay for it.
Data centres are better than hosting in your office but are vulnerable to outages as many people know well. Our last outage was because some idiot (a data centre engineer?) switched off power thinking it was only going to affect someone else's rack. They may have fire supressing gas and the best security system, but there is no technology to prevent the employment of idiots. And there will always be idiots.
come on guys....it was a fire and at end of day its better keepin generators off as instructed by a fire department. servers are safer off and isolated while fires and cabling is checked over
While they save up on their electricity bills, small businesses are going to pay heavily!
all our 3 servers are down!
they will survive! but will we???
...Would have installed a firewall.
No 'coat' tag because I can't figure out which to click with the blackberry browser.
Still offline.
You never know how much you miss something until it's gone. Hope they get everything back, and soon...
Cheers
"I once had a tour of a data centre in the UK and was shocked to find their fire extinguisher system was "water sprinklers""
A dry-pipe water system, Vesda particulate detector, and continuous staffing are the usual approach.
The Vesda system sets off an alarm and a tech with a fire extinguisher goes hunting for smoke. This allows the usual sort of computer-based fire to be handled with little damage to surrounding servers (usually they just get the power dropped as the tech drops the rack's circuits prior to removing the smoking gear, taking it outside, then opening the box and applying the extinguisher).
The water system is for the last resort, usually from a fire in another part of the building reaching the computer room. It's not unreasonable for the insurance company to sacrifice the computer room if that saves the building -- anyway, they are paying for the damage to both so it's their call.
Gas got unpopular when CPUs got small, numerous and hot and computer rooms got very, very large. If you think through the consequences of a cooling gas hitting a modern hot CPU and the problems of venting released gas from a large space you'll see the problems.
Fixed powder-based systems aren't a good fit to computers. An aerosol-based system would be a better fit.
Despite the double entendre in the title, that with which I need help is replication of a MySQL database over two servers at differing locations. I have read the MySQL manuals, but I would appreciate a pointer to a tutorial or a book which explains the procedures in more detail. Currently I am working with a 1 gigabyte DB, and I would like to mirror or replicate it so I don't lose everything next time a server-farm disappears...
I think you guys are missing the point, it's not about colocation or backups or power or any of that shite. It's about b3ta. What if there's no backup to the b3ta archive? It'd be like the library of Alexandria over again. 5:30AM on Monday and still nothing. I'm not a religious man but here goes.. Allah wu Akbar, Allah the digital, the compassionate please restore the purple cock and domo.
I'm in the office, and can't read the QOTW archive. I also have yet to see a magenta cock today. This is wrong for a Monday.
Why have you bothered to include the link to b3ta when you know fully well it's unobtainable?
That data centre with water wouldn't happen to be in Edinburgh would it? I think I've been there too.
"Allah wu Akbar, Allah the digital, the compassionate please restore the purple cock and domo." .... By Seán Posted Monday 2nd June 2008 04:27 GMT
Amen and Hallelujah to That Passionate Restore Point of Immaculate Imperfect Relevance, Seán.
Love ur dDutch. .... Real Get SMARTer IntelAIgents.
Here's a Virtual IntelAIgents Swap Shop/Treasure Vault ....... http://www.ams-ix.net/
I'll get my coat ....there's a CAB AI Called.
Work's blocking the Internet Archive's Wayback Machine; I can't even see if they've got older versions of B3TA squirrelled away anywhere!
This is ridiculous. Surely The Planet have backups, disaster recovery, that sort of thing?! Can you imagine if the Emergency Services said "Well we can't actually man the 999 phonelines 24/7/365. We'll need a week off every so often but we'll compensate anyone financially suffering from our unavailability"? Well this is even more serious! 9.15 and B3TA is still down, people!
... has a slightly unfortunate link title to it, "How fast is youre network?".
I don't know about mine, but parts of his network hit about 50mph recently!
Houston it's those damned insulation eating ants that did them in I bet.
I was 3/4 the way through coding a P2P message board that replicated the B3ta messageboard and would have mitiagted this, but then I gave up through lack of interest. http://sourceforge.net/projects/b3ta
Will they never listen, think of the children, apologies for length or lack of.
Productivity across UK offices must be at an all time high for a Monday morning this must be some sort of xonspiracy. Helicopter for obvious reasons.
There's a temp board set up by Rob here
http://forum.robmanuel.com/viewforum.php?f=3
An explosion of the local utility transformer took Rackshack's main DC off grid for 4 days a few years ago. Not a minute of downtime was experienced by 17,000 servers.
The subsequent write-up of the event showed both an amazing amount of pre-planning that initially kept everything going and fast adaption to cope with unexpected consequences to keep it going. A long list of lessons were learnt at Rackshack. Were these all passed on to The Planet when it acquired them?
And anyone who has a mission critical server without a geographical seperate backup - presumably doesn't understand the concept of backup - or why you have a minimum of two DNS. When those phones start ringing I hope they say "You are fired!". Putting client's businesses at risk (like no email?) is just darn unethical as well as bad business.
O look forward to hearing any excuses ... from £60/month for a deicated server phrases like a pennyworth of tar come to mind.
It wouldn't take too much for a co-operative to be set up distributing activities between a predetermined number of other hosts until the crisis is over.
Incidentally would I have a legal case against b3ta for making actually have to do some worth on a Monday morning and the mental anguish caused by this?
"Allah wu Akbar, Allah the digital, the compassionate please restore the purple cock and domo." .... By Seán Posted Monday 2nd June 2008 04:27 GMT
Ensha Allah.
Or, in layman's terms: the computers were built thanks to Allah, the data was put there by the hand of Allah, the colocation duplication systems were denied by the mighty will of Allah, the fire was started by the great and merciful Allah and the DNS servers are still down thanks to the estemed and bountiful Allah. Allah be praised - and the rest of us thank fuck it wasn't organised by LizardGov.uk otherwise the data centre would probably still only be half built, at half the original spec for quadruple the cost.
Can we go to stoning now?
Especially ones with "I would like to provide an update on where we stand following yesterday's explosion ..."
Well I guess thats why we don't have our power transformers indoors then!
When they blow up they can do it peacefully in the car park while the UPS kicks in and prepares the generators for taking the load. When they kick in you see a mushroom cloud of diesel smoke, god knows what people think has happened when they see it!
I guess this is just a bad luck story, I can see they are working hard to repair this and get them back online. Would you like to be the one to reboot 9000 servers lol.
Definitely think they could have had a better disaster recovery plan in place. Seems like they only had a basic one and thats it....
The way to make it most resilient is to have two buildings kind of co-located (same business park etc) but not physically adjoined. So if one gets nuked the other can continue.
On the bright side I think they must have saved a bit on the leccy bill.... oops wheres me coat
...still no b3ta. I may have to go out in the fresh air. Nice to see a few b3tans on this comments page here, though.
If our El Reg moderatrix will permit it (pretty please, Sarah), may I invite Dr Trevor Marshall and other interested parties to join us for discussions at:
http://www.opensolaris.org/os/community/ha-clusters/
and/or
http://blogs.sun.com/SC/
and look particularly for entries related to the Geographic Edition.
So is this why the Heathrow security guy made that fella take his Transformers T-shirt off?
From the seriousness, that no one was hurt or injured. :)
Productivity has increased tenfold. I hope b3ta/links is back on line soon. I have a rather humorous clip of Rick Astley to post . . .
I've done far too much work today, and not enough skiving
Rob's emergency board isn't quite providing the same fix
lo, the greebo warrior
why I spent the whole of Sunday furiously clicking 'refresh' on b3ta.com to no avail, failing to notice anything else around me, or remember to eat.
It's the /talk people I feel sorry for - at least the rest of us can look at the pretty pictures on 4chan.
(Peregrin)
I have nothing hosted with them, but it sounds like they were doing fine until the fire dept. forced them to shut down their generators. True, it's not as good as total redundancy, but again, it sounds like they could have coped. Probably the generators weren't even a slight part of the problem--just playing it safe.
From the status page/forum post:
1337 User(s) are reading this topic
Thanks for the compliment, The Planet!
Mine's the one hanging on that wall over... oh, bugger.
The systems commonly used are of the HI-FOG mist fire suppression type. The pipe work and nozzles are often mistaken for 'sprinklers' but in fact discharge a very fine mist that puts out the fire and is safe for humans and the hardware. Very common for DC and Telecoms applications.
Gas discharge systems are expensive and the older CO2 fueled systems can be lethal to humans in areas where there's no ventilation.
Paris....'cos she enjoys a good sprinkling every now and again.
Sign up, sign up for Blocks and Files, The Reg's weekly storage newsletter
