The data center at online brokerage house TD Ameritrade in Jersey City, New Jersey handled more drama yesterday than the usual ups and downs of the market, after a smoking power supply in a blade server caused fire suppression systems to kick in and the building to be evacuated. This is not a situation that any IT vendor likes …
Fire at 41,000 feet
My first flight on on NASA's Kuiper Airborne Observatory was interrupted by a smoking HP power supply. The telescope data system was all HP equipment (2100 mini-computers or earlier) and something shorted out during the flight. As the smoke increased it started to get everyone a little worried. Fires on airplanes are not fun. Everyone went back to their seats and someone grabbed a fire ax and started looking for some equipment to attack. The system had a ground fault protector that finally tripped and shut down the whole system. We had to land with the telescope enclosure open (a huge open hole in the side of the airplane) which had never been done before. Not exactly the best flight to start out with.
This is a link to some photos I took on the airplane:
One possibility you omitted to mention...
>"The smokey power supply at TD Ameritrade might represent a different issue with the power supplies, or it may be related. "
Well, or it could be a one off failure in this particular unit and might not represent anything at all.
Thanks for the link and pictures - some interesting stuff there :)
Also it's good to see that there are actually still some normal decent people with interesting/useful comments and info, reading/posting at the Reg - I'd begun to think the Reg was only occupied by sleazy creeps, given some of the sleazy comments that many *other* people make on most other articles here. Anyway thanks again :)
or a bug crawled in to it? never had it happen to a powered system, but it does give a nice bit of smoke when i've pulled systems out of storage and gone to turn them on...
How does it work?
From what I can tell about major data centres, all access is heavily secured (finger prints, biometrics, etc), and several even have one-person+server air locks into the equipment rooms.
So having said all that, how do the firemen/women get in there?
I imagine there are dry risers so no need to trail hoses through the air locks, but they're not eactly conducive to rushing!
Squash this rumor!!!
It is not true that HP Blade Servers are going up in smoke all over the country! Not true!
Hows about betting it was a customer caused issue?
I dont think HP blades would be were they are today if they had an inherant design flaws. I'm betting it was either the first time the power supply was run up and it was a faulty unit (standards slipping with QC HP?) or someone was being a spanner!
Oh and before IBM get all cocky ask them to come and explain to me why one of their chassis was on fire in the DC the other day? Having run HP and IBM blades I can safely say that whilst neither of them are 100% foolproof the HP is a damm site better unit that at least makes more sense than the IBM and based on my experiance has a lot less points of failiure on it (not that a decent systems architect uses just 1 chassis in a system!). Give me 1 minute with an IBM chassis and Ill show you at least 5 ways to degrade service significantly and usually bring the chassis down!
From what I can tell about major data centres, all access is heavily secured (finger prints, biometrics,
Go to the data center for bank of America in SF.ca. Its key card only. This date center spans 4 floors. Sure it has the hand print reader, but I've never seen it work. Every one just used a key card.
@AC [28th March 2009 22:32]
"I dont think HP blades would be were they are today if they had an inherant design flaws. "
Individual parts in products change from time to time and it's possible that HP picked up a batch of power supplies that are slightly less tolerant than normal.
I recall having a batch of Compaq (remember them?) PCs whose power supplies were prone to going bang and filling the room with foul-smelling smoke. We had a few go while commissioning them and more than a few after delivery. One of our more elderly secretaries got quite a shock when her new PC exploded and even more of a shock when the replacement unit we brought her did exactly the same thing. She thought we were trying to save the organisation the cost of retiring her...!
The worst part was it took HP over 2 hours to get there
Every other vendor was there and had their stuff up and running before HP even showed up
"...I'd begun to think the Reg was only occupied by sleazy creeps, given some of the sleazy comments that many *other* people make on most other articles here. Anyway thanks again :)"
I resent that. We are at least mostly sleazy creeps. That makes sure our topics are off-topic, flaming, intolerant and generally unacceptable; we live up to our reputation. What the heck is your problem, anyway?
- Nokia: Read our Maps, Samsung – we're HERE for the Gear
- Kaspersky backpedals on 'done nothing wrong, nothing to fear' blather
- Episode 9 BOFH: The current value of our IT ASSets? Minus eleventy-seven...
- Too slow with that iPhone refresh, Apple: Android is GOBBLING up US mobile market
- NASA to reformat Opportunity rover's memory from 125 million miles away