Thundering gas destroys disks during data centre incident A few weeks ago in a Sydney data centre, something started making smoke. As its designers intended, the centre's reservoirs of inert gas started venting to stop any fire spreading. The smoke was contained, but not long afterwards tenants started to complain that some of their disks had not survived the incident, which was odd …
Saw this myself last year, we had a customer run a fire alarm test of some kind, fried a huge number of SAS drives. From our systems around 50% of the SAS drives were completely toast, unrecoverable. The SATA drives were fine, and I understand it was a similar story with other vendor systems too.
Most modern disks will get damaged if you significantly change the air composition and/or air density (they all have air vents). The head design relies on aerodynamics to maintain the correct distance from the platter. Air density affects that quite a bit.
Just ask any Colorado resident - the stats for disk reliability around that parts are way off compared to stats in normal places around sea level. Some models which fare perfectly fine elsewhere regularly fail in 60 days or less.
So the fact that someone releasing fire extinguishing gas (which is quite dense by the way) has changed the MTBF of drives does not surprise me in the slightest. SAS is likely to fare worse too as it spins at higher revs and is engineered to lower tolerances.
Vibrations from "whistling", yeah some other time, b*ll*cks... The "storage engineers" that were asked need to get a clue what is a modern disk, how it is designed and operates.
Lived in CO most of my life. Never experienced or heard of the air pressure negatively impacting hard drives, including friends who work at IT installations at CU Boulder and who live/work at ski resorts at 9000+ ft. (Also, airplanes are usually pressurized to around 8000 ft equivalent. Haven't heard of any hard drives crashing when used on an airplane.)
Short life! In the mid-80's (when dinosaurs still roamed the data center) DEC had hideous problems with their RA80 14" winchester drives. Apparently nobody had taken into account in the design cycle, that some highways in the US (Colorado in particular) reach altitudes of 11,000 feet / 3350m. As I recall, the disks were in sealed units with slightly pressurized gas; the seals failed at something like 9,000 feet / 2700m, the gas leaked out & or air leaked in, bad things ensued. Major sadness in the sales, services and support community. Took a bit to suss out that little mess.
I remember working for a large mainframe customer in the eighties where, over one weekend a large number of 3380 hard disks mysteriously crashed. The culprit was later found to be the use of oil based paints for renovating the data centre. Disk heads float on air, air + thinners = head crashes all round.
Upon analysing our old Halon systems, prior to retrofit, we found a teeny weeny flaw. Before removed the old kit he guys looked at the system (along with some internal guys that know fire suppression systems way better than me), realised that the release would happen so fast (they estimated less than a second or two), that you wouldn't have time to die from asphyxiation, as the floating floor tiles being blown several feet into the air are far more likely to kill you.
Email that anecdote to Simon Travaglia. I'm sure he would be glad to hear of another way he can remove a boss once he is locked in the server room.
Speaking of Simon, where is he? I'm missing my monthly dose of psychosis. If I start suffering withdrawal symptoms, it will not bode well for the muppets I have to endure at work.
I would not have believed if if I had not seen it.
Note it's not the level it's sudden (and steep) changes in noise level that does it.
BTW. From nozzle theory almost any kind of pipe that has a gas tank with more than about 2 atm pressure difference between it and the exhaust will give an exhaust at the speed of sound.
Low volume (70 psi) pneumatic industrial control systems leak like this.
It's very loud.
'Vibrations from "whistling", yeah some other time, b*ll*cks... The "storage engineers" that were asked need to get a clue what is a modern disk, how it is designed and operates.'
What won me over was your carefully researched analysis and the trouble you took to substantiate your hypothesis.
Have you watched the YouTube video linked above, BTW?
That guy is a lot more slapdash in his scientific approach than you are. He just uses graphs and stuff and doesn't even bother putting his shouty words into this comment stream.
Direct lightning strike on the roof triggered the halon system. Even though the room was supposed to be suitably vented, the overpressure from the gas shifted some of the interior walls as well as destroying the false ceiling. Not at all surprising that some disks suffered, that sort of step pressure change could easily cause "flying" heads to bounce enough to polish the spinning rust.
Remarkably, between a good UPS that absorbed the electrical surge, and proper backups, there was surprisingly little disruption to normal operations.
Pressure changes can do all sorts of unusual things. In our Belfast office, 30-odd years ago, a car bomb in the street oustide shattered windows and broke internal walls. The offices had a good supply of VT100 and VT220 terminals. All the VT100s survived, all the VT220 screens were OK, but many of the VT220 keyboards didn't work afterwards. We never did identify which component of a keyboard was that pressure sensitive!
--- Customer support here, good morning. Yessir, bit of an outage. No, nothing serious Sir, just another car bomb. We'll be back online in a few mins, just wiping off the glass shards now.
You may laugh, but the explosion happened at 4pm on a Friday (we were in the pub, the building having been evacuated). The emergency plan went into action, some empty office space was acquired and cabled over the weekend. DEC delivered a lorryload of new equipment at 9am Monday, and by lunchtime everyone in the affected office had a desk with an operational phone & terminal. One customer who phoned expecting that their scheduled meeting was cancelled was more than surprised to be told that everything was still on track.
DR planning works!
Mind you, for weeks afterward we were opening files that had been inside steel filing cabinets, and shaking the broken glass out. Made us realise what that trite BBC phase "some people were cut by flying glass" could really translate to.
Posting anon for good reason, I work in enterprise storage.
From an enterprise storage support prospective, I have seen this a number of times.
In most cases the drives/data were recoverable, in a couple it was not.
A lot of contractors seem to be putting together way too big of a system for the room (more profit for them). Also remember you have to "fill" the room quickly, you have sonic issues, you have a shock wave, you have a rapid change in room temp (it cools quickly), in addition to the pressure changes.
It's gotten to the point that I would highly consider a well planned dry pipe water based system with proper drainage vs a gas based system. The only equipment to get soaked will be the equipment in the area of the fire (since water based systems only activate the head under the high intensity heat source), and you don't have to worry as much about the rest of your equipment.
A sudden pressure change from these gas systems could perforate both your eardrums. Having worked in one of these centres I worked out how far I could run in the 30 seconds the system could be held off for. I reckoned I could just get safely clear before the gas went off because the stairs were a lot further away than the lift.
If you work in one of these environments and don't run regularly - take up running now, because nothing can replace your hearing.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
