'Powerful blast' at Glasgow City Council data centre prompts IT meltdown The catastrophic service outage at Glasgow City Council's data centre, caused after its IT systems servers were taken down by a fire suppressant accidentally going off, is continuing to cause widespread havoc for staff and the public. The embarrassing blunder was caused by a faulty air conditioning unit setting off its fire …
This post has been deleted by its author
I like how the picture on the article is a f****g kilt. Sod all to do with the story beyond the fact it's in Scotland, they couldn't even be arsed bringing out the Glaswegian stereotypes. Scotland = Shortbread Tin Highlander.
Anyway, it turns out that Sheffield Council are having some problems with their IT systems. Here's a picture of what that might look like. A spokesman said "Lor luv a duck, I know I'm not even the right faacking stereotype, but it's all daaaarn saaarf, innit! Root-te-toot, apples and pears, etc."
Pictures? ... fortunately the "banner" pics in all reg articles follow teh same URL pattern so its simple to use a custom AdBlock rule to hide them - been doing this since they arrived and can't say I've ever felt I'm missing anything. Also added another rule to hide those stupid "badges" and along with all the other stuff that gets blocked makes ElReg a much more pleasant read!
"I like how the picture on the article is a f****g kilt. Sod all to do with the story beyond the fact it's in Scotland, they couldn't even be arsed bringing out the Glaswegian stereotypes"
I agree completely...
This picture is far more appropriate to Glasgow and its people
This post has been deleted by its author
Sometimes they like to fuck up their filing systems and then spend a decade pursuing whoever pointed out the mistake in order to extract large sums of money, at the same time as they're breaching the data protection act and tossing around threats of arrest and imprisonment.
All because I pointed out that no, I'm not my dad, please correct your records...
Maintain roads. Schools. Social care. Pre-school education. Children's/Family services. Youth centers. Youth justice (eg. secure accomodation), street lighting, Social housing. Dealing with homelessness. Museums and galleries. Recreation areas (parks, leisure centres, sports facilities, playgrounds etc...), tourism information and promotion, libraries, cemetaries/cremation, trading standards and community safety, environmental health (food safety, pest control, pollution control), licensing (alcohol licensing, taxis, events), garbage collection, planning control, regeneration, fire services, coroners, registration of births, deaths and marriages, and election admin...
So yeah, what do they do for us hey?
Some clarification needed. Most IT kit is largely air by volume. And when the system goes off, presumably the air in the data centre has to be displaced to somewhere, otherwise the pressure must build up.
What kind of rate of pressure rise are we looking at? How is it vented?
Okay, imagine a room three metres wide, three metres long, three metres high.
That's nine cubic metres by my reckoning.
The room contains three cubic metres of kit. The fire company has calculated the quantity of FM-200 or whatever it is to fill the remaining six cubic metres. If you add more kit and the gas goes off, it's got to go somewhere. So either the roof pops off or the walls shift.
That's GCSE physics surely. That's my understanding as a humble data centre BOFH not a million miles from Glasgow, I'm not a gaseous fire suppression engineer.
P.S. The SNP will most likely blame "Westminster" and/or David Cameron. Still, we've got road signs in Gaelic, and that's really important.
Not quite sure what happened. And what do they mean by a faulty AC unit? Was it faulty as in going to catch fire!? If there's a VESDA (Very Early Smoke Detection Apparatus) system then that would be set off by the AC if it was producing prefire vapours (that's the whole point of VESDA) it detects "fires" before they turn in to fires! But normally you wouldn't want this to set off your suppression system, initially you'd want it to display an alarm on the fire panel so it could be investigated, not trigger the house alarm and discharge your gas! Or was it faulty and something happened like it produced a load of dust or something else that could also have been detect by the VESDA? This is why its important to remember to disconnect your suppression system BEFORE you let anyone in the DC to service kit etc! But whatever it shouldn't have set the gas off and the release of gas shouldn't have caused the damage that it did!
The story mentions failed AC equipment, but no real info on the exact nature. If my experience, AC failure units pissing water down the walls of a server room, and it cooking at 70C until the servers auto-power down is the main symptoms you get.
I dont see why you need to calculate area and kit volume etc. I would just
- position inert gas dispensers at one end of room
- put hole at other end
-in event of fire blast inert gas into room - allowing air to escape through hole
- when sensors reveal air gone and room full of inert gas, switch off.
Also if you have 24 cu m. of air and you double it with a gas, depending on what type, you get 2 atmo of pressure or 28.8 lb/sqr inch. while this might blow out a loose window pane it should not be enough to blow out walls. In a well designed data centre.
Yes hurricaines and tornados do blow down building using less pressure, however they have wind shear and wind velocity to help.
Gases can compress pretty well, and if your walls blow out due to a small change in the contents of a room, then those walls were made of paper.
The building shaking is likely as a consequence of the fixed equipment venting all that gas in a matter of a couple of seconds. There's gonna be an equal and opposite reaction to the gas being deployed somehow...
P.S. The SNP will most likely blame "Westminster" and/or David Cameron. Still, we've got road signs in Gaelic, and that's really important.
No need, Glasgow City Council is Labour run (and has been for years).
Also, the Gaelic tends to only get added when the signs are getting replaced anyway.
"That's GCSE physics surely."
- It it, aye? Given that the GCSE isn't generally taught or taken in Scotland... That sounds more like "Standard Grade" or worse still 'National 4/5' 'Marfs'...
If you've ever had to stand in front of class full of new NQ or Even HN1 students in a Scottish FE college and try to teach them anything remotely technical you'll know what I mean. At least in t'old days nobody thought the subjects they failed at school 'qualified' them for anything!
I found this musical interlude...
https://youtu.be/GG_Vdh-WuGY
...Which seems highly appropriate when considering activities at 'Castle Greyskull' (Glasgow City Chambers). "A gaseous fire suppression engineer" - as opposed to solid or liquid - an actual ghost perhaps? - Could dance with the skeletons hidden in all those basement cupboards...
Being Glesga though it'll probably just be a fat bloke with gall-bladder problems.
This post has been deleted by its author
Jeez, what kind of gas-based fire suppression system shakes buildings and seriously damages equipment?
Well, as I commented in a Contingency Planning report for xxxxxCo some years ago "I have been unable to confirm the Ops. dept. claim that soldered 15mm copper tubing can handle the release of unregulated CO2"
[missing lightbulb icon]
.... but when halon was banned for refrigeration, some cowboys started using butane (works very well unless you get a leak!).
Surely no-one would replace fire suppression gas with butane....
I might be wrong, but I suspect the council are looking for a reasons why this caused systems to die and building vibrations are a logical quick answer.
From experience...ahem... I might suggest it was the noise of the gas venting which created vibrations in the air which took out the disk arrays. Other people have had the same issue...
http://www.theregister.co.uk/2014/02/04/loud_data_centre_gas_release_sounds_harm_disks/
https://www.youtube.com/watch?v=tDacjrSCeq4
G-Cloud??
They used Glesga Cloud! - A smoke-filled daylightless area (formerly the Janitors store ) in the former Primary School next to the Red Road Flats; where Nero Mathieson's Denzians gathered to stoically collate floppies... Wonder if anyone thought to tell them they were blowing the place up? - As in actually demolishing it!
Having worked in local council's for many years the loss of IT systems for a few days will have enabled most workers to actually get their jobs done without all the nonsense enabled by the computer systems.
Of course once it's back up then the managers and bureaucrats will want all their monitoring and logging done which will add to the backlog of things to do in the new year but it will be paradise while the computers are out.
I happen to know GCC IT are a pile of poo, but I'll not quote specifics as I've alerted friends who actually worked there to this story and they can choose to chip in.
I used to work for a similar nearby council whose Finance Director didn't trust mag tape back-ups not to degrade, so all council records had to be printed out and stored in a huge storeroom under the town hall. Yet nobody could ever have found any meaningful data in that huge pile of flammable paper, and when I checked all the older records ink had faded away anyway.
The reason we have Scottish councillors and council officials is to keep them busy and away from more important careers like dog-walking.
Seeing as emergency services weren't affected, I'm not sure what all the hoopla is about.
Just because a DR/BC plan is in place, doesn't mean its been designed for maximum uptime. It could have been designed around minimal cost, or somewhere in-between. They do mention they have backups. Seeing as gov't tends to burn money rather than make money, their planning around uptime is likely going to be considerably lower than a profit oriented corporation where downtime is actually reducing revenue, or reducing the twattering customers can do with their thumbs.
Assuming that the mentioned services they can still perform manually comprise their list of 'critical services' then it sounds like they're doing ok.
As for the boom/shaking; this should be expected if a halon system went off. A canister went boom when I dropped it on a concrete floor, and I went about 2ft up in the air, so I'd guess if they all went off in an enclosed environment, you might get some feedback.
It's not only ironic that the disaster suppression system going off caused a disaster, but it caused more damage than it's supposed to prevent in the first place.
If they'd really had a fire, there'd have have been a small amount of fire damage and the rest of the kit blown to bits instead of saved!
This is Glasgow City Council's Forth Road Bridge. The parallels are there - reduced maintenance, reduction in failsafe capacity and redundancy in a short-sighted effort to reduce costs following the transfer of responsibility of running the system from several expert external contractors to an arms-length quango. - https://www.access.uk.com/
Is the journo that wrote this a fucking retard ??
"Powerful Blast".... of fire suppressant
A bloke in a kilt, as it is Scotland.
What a complete dick.
Glasgow City Council, where is your DR plan, as a statuatory body, you should have a good, documented and fully tested one. Hopefully, it will interefere with your Soprano's style bus lane camera operation.
"Perhaps more problematic is the fact that there was no backup provision at all for the majority of systems."
AND
"All our data was backed up and the business continuity plans in place meant those services were manually delivered."
So basically another half-arsed, "shit never goes wrong why spend money" security/DR plan then! So no live fed DR site for a critical city infrastructure. We'll just back up the data but we've no provision for a DR situation in which to restore the sodding critical data, not even a standby agreement with our kit provider to supply XYZ kit in an emergency. Cue frantic calls to IT staff and IT purchasing dept to try to arrange a site/some kit in a few hours, only to discover the only spare SAN array like the one we need is sitting in a customs somewhere in a Dutch cargo port.
IT: "Can we have a proper DR plan put in, here's the details of what we need."
Management: "HOW FOOKING MUCH?!!"
IT: "OK, we have 'PLAN B', it's dead cheap and we can do a few backups for a couple of quid but it won't work very in an emer...."
Management: "Yep, that's the one! Make it happen!"
"All our data was backed up and the business continuity plans in place meant those services were manually delivered."
No, business continuity means business continuing not stopping and hoping the backups worked. Here's a design based around a single point of failure. A second room with a copy of the kit too much to afford? A backup system on AWS or Azure or Rackspace or anywhere that can be enabled on-demand? Let's hope the architect's other designs are being reviewed.
Especially since the shift to PMR hard disk recording, the gas release can be very very bad for hard disk drives.
Gas fire suppression tends to impact equipment in two ways, especially if over sized. First is it causes a very very quick cooling of the room (over 30C drop can happen). While equipment tends to do well with cold temps, it does NOT do well with a very quick drop in temp. The second is that the gas release is VERY loud (over 200 decibels is not uncommon), combined with the required VERY loud (over 60 decibels) alarms that must go off prior to the gas release.
Please review your fire suppression plans, IMHO dry pipe water may actually be safer in the DC vs any of the gas options. In most systems sprinklers only go off over the "heat" and not the entire room.
Anon, as I work in the storage industry.
One alternative I've seen elsewhere is a set of individual fire-extinguisher "tubes".
You put one inside every piece of kit the could start a fire.
If it starts heating too much the tube melts at the hottest spot and poof! the fire is doused.
There's a sensor at the end of each tube to detect a discharge.
It probably damages the kit it's inside - but that was on fire anyway.
Quiet, simple and relatively cheap.
Couple with a zoned sprinkler system over the AC and I suspect you'd never lose more than the box that went on fire.
IMHO dry pipe water may actually be safer ...
plus ca change!
that was also the consensus (tho' rarely implemented) opinion 20 - 25 years ago.
... pull all the boards, detergent wash, distilled water wash, warm air blow dry (yup, hairdryers) - reportedly better than 90% recovery.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
