Feeds

back to article BOFH: The Great Patch Mismatch

"It's just a minor ROM patch." the service engineer bleats "It'll only take five minutes." "Yeah... Nah," the PFY says. "It's minor - just addresses a couple of memory leaks and and cookie issues in the web interface." "Yeah. Nah," I repeat. "It's just the interface - the UPS will be completely unaffected!" "Nope," the PFY …

COMMENTS

This topic is closed for new posts.

Page:

Pint

Needs more KZZZEEERRRTTT!

Ah Friday. BOFHday...

2
0
Bronze badge
Headmaster

Don't think we really needed the footnote... *

6
0
Holmes

I concur. If you need the footnote, you won't get the joke.

4
0
Silver badge

A slippery slope...

If you're going to explain what a Halon system is, where are you planning to stop?

Relays? UPS? Patch panel?

0
0

RE: Don't think we really needed the footnote... *

You're beeing way too polite with the author on this one... ;)

1
0
Silver badge
Thumb Up

Brilliant

Thanks! Makes for a nice weekend :)

3
0

Ah yes...

... the good old update that definitely, positively, absolutely won't need you to restart your device... except that it always does.

If anything/anyone promises me that a reboot won't be needed I automatically assume there will be at least two forced reboots before everything is working again.

Also, the more mission critical the device being updated the more likely the update is to go wrong.

17
0
Anonymous Coward

>My assistant and I subscribe to the belief that if it ain't broke it don't need fixing

So in any attempt to find out where these two work we can exclude Ferrari

6
0

Real BOFH fans...

Real BOFH fans don't need to be told what the halon system is...

24
0
Anonymous Coward

Re: Real BOFH fans...

Judging by some of the comments on other articles I get a powerful impression that the only relationship some of the commentards have with computing is one to a distant relative, and even then only by marriage, who at one time browsed round a PC World store becuase it was raining out.

6
0
Gimp

Re: Real BOFH fans...

some of us know even less than that and use macs with virtual PCs...

shudder

2
0
Silver badge

Re: Real BOFH fans...

Agreed, if you don't know what a halon system is, get a job not in IT. But, hasn't halon been banned for some years now? IIRC it was in the early 2000s when we were busy exchanging halon with other inert gas - the "oops, we just had a little fire in the data centre and the halon has been released automatically" kind of stuff.

0
0

Re: Real BOFH fans...

It has been banned in new installations since then. Existing installations are grandfathered.

2
0
Anonymous Coward

Don't talk to me about...

... UPSs - my desktop APC one needed a new battery - which could be replaced without powering down. Great.

But to make it realise the battery is now ok, and stop the bloody thing beeping at me every few hours I have to power cycle it.

Fail.

9
0

Re: Don't talk to me about...

APC? If it is the entire unit will probably develop a mysterious fault where irt just decides to randomly switch itself off wihtin about 6 months of changing the battery.

4
0

Re: Don't talk to me about...

Not as bad as having a racked APC UPS with what looks like a serial port on the back that's magically wired such that if you dont use the special APC cable and instead use, I dunno, any other bloody serial cable, you trigger an immediate shutdown of the UPS because they've done something bloody stupid with their NotASerialPort and helpfully decided not to put any warning stickers near said port...

9
0

Re: Don't talk to me about...

I seem to remember that they short two of the pins in their un-serial cable.

1
0
Bronze badge

Just had to update an update of an update of an update that "would fix our issues".

Turns out the expensive Management Information System that our school runs needed an update ages ago. We didn't apply it because we didn't need any of the changes it had. The next week, we were issued with an emergency update that fixed the corruption of the main databases caused by that update. We applied it weeks later when we had heard it was safe. Surprisingly, the new update backs up the old database before it touches it now (seemed kind of a big omission in such a large piece of software, but what the hell do I know?).

So we do that, but that needed us to install IIS and ASP on the server that runs it, because the update-to-the-update also migrates you to their new "web-based" infrastructure even if you don't actually use it. Given that that server runs NOTHING else, we let it happen. Then we updated again to test the new HMRC PAYE tax notification system which was touted as a feature and we were on the trial for. Turns out that just gives SOAP errors because some silly sod forget to make the update-to-the-update-to-the-update actually activate ASP functionality (and given that the update came with install instructions that don't mention anything but installing IIS in its default config, that was kind of an oversight). We just had them log in to work out WHY it didn't do that and apparently there's going to be an update-to-the-update-to-the-update-to-the-update to fix that issue.

I wouldn't mind, but we didn't have any issues to start with, only narrowly avoided complete database corruption by being stubborn and refusing to update, and we ended up losing the functionality to submit data to HMRC. Luckily nothing important until April but that's a pretty damn unusual testing procedure they have there. Oh, and in the process we annoyed the guy doing school reports because every update wiped out any school reports that had been taken offline and he was testing them for later in the year.

Years of experience have taught me: Never update automatically. Never update immediately. And, if nothing's broke and it's not a "frontline" machine, never update. And certainly never update with a way to get back to EXACTLY where you were before the update.

Applies to everything from modem / router / switch firmware up to and including network-wide application software.

28
0
Silver badge

Updating ... and why not to

@Lee Good post, frustrating when users then accuse IT of being slack/tardy for not rolling out updates without giving them a chance to be proven.

3
0
DJV

Updates

Yeah, I never update Ubuntu servers any more. Have been bitten once too many times (i.e. twice). Far safer to install a new version from scratch!

1
0
Vic
Silver badge

> the expensive Management Information System that our school runs

I am *so* glad I don't have to deal with SIMS any more...

Vic.

0
0
Bronze badge

Nearly, but SIMS isn't the only MIS that suffers these issues.

Hint: Private school software a company in the West Country.

0
0
Silver badge
Thumb Up

Halon Gas

When you absolutely have to kill every fucking problem in the room!

12
0
Flame

Re: Halon Gas

When all you have is Halon Gas, everything looks like a fire.

17
0
Anonymous Coward

This brings me happy memories

In the good old days we had a detector break and set off the zone one alarm - the fire brigade arrived and we pointed out the problem, they duly removed the faulty pot and tested it in zone two - oh how we laughed when we they set off the Halon, it seems that if two zones go off then so does the gas. Even in all that equipment the fire brigade can run like bolt - who knew

6
0
Bronze badge

Re: This brings me happy memories

I never got the point of halon.

Let's prevent deaths from smoke inhalation by killing anybody who might be in the room wanting to breathe.

Sure, halon leaves your equipment undamaged, but if you're important enough to worry about that, you have redundancy elsewhere anyway. Just install a separate sprinkler in those rooms and insure the servers if it comes to that. And then no H&S and legal nightmares, no extra equipment like masks and breathers, and no special installation and precautions necessary.

I never got why computers, of all things, should take priority. *Especially* if those computers are supposed to save lives themselves (e.g. 999 datacenters) - then they should be highly redundant and distributed anyway.

1
7
Pint

Re: This brings me happy memories

I was told, as a young PFY, fleshy's can be found anywhere, computers are expensive and the data on them even more.

Our halon system went off due to a fault, we opened a few windows and doors and evacuated the building until the fire brigade came by and checked it out to be safe. I was working on a big industrial chemical plant at the time, they were relived that the escaping gas was inert. The day the OTHER siren went off now that was a different story all together.

7
1
J P

The point of Halon

They didn't just use it on computers. Libraries loved it; apparently priceless medieaval manuscripts and irreplaceable first editions respond slightly less well to sprinklers than do server racks (the concept of double redundancy isn't really compatible with *unique* archive material). The alternative gas systems are I understand considerably less effective, so more books will burn while it takes effect. IANAL*

*I Am Not A Librarian

10
0

Re: This brings me happy memories

@Lee Dowling

I've a bit of time, so I will explain it to you.

Fire requires three things: Fuel, Oxygen and Heat. If you have a fire, removing one of those things will kill it stone cold dead. Water acts to reduce heat. All other fire prevention works to remove the Oxygen.

In the event of an electrical fire, you do not want to use water: Water and electricity do not mix. So you use oxygen suppression.

Halon is an oxygen suppression system just like CO2. Being a gas, it can get inside the servers to where the fire is (usually the PSU) and put it out BEFORE it gets out of control and sets the entire building alight, potentially creating clouds of toxic gas, putting neighboring buildings at risk, other lives at risk etc. Other systems such as water, foam and powder, would only coat the outside of the case and as such would be very ineffective in suppressing the fire. Indeed, in the case of water, it would only make things worse.

Cutting the power is usual practice, too, but that is not always sufficient to stop a fire.

Halon, CO2 and other gas suppression systems work best in a confined space. All will suffocate anyone caught inside that confined space as they are removing the Oxygen living beings, such as humans, require to live.

As such, any area where Halon, CO2 or other oxygen replacing fire suppression systems are installed, will have things like oxygen masks, big red buttons to delay the release of the gas, release locked doors and so on - so if someone is caught inside the room they can act to protect their own lives. If they happen to be unconscious on the floor because they just electrocuted themselves and started an electrical fire... well, that's called Darwin's law of selection. Same as if they were rendered unconscious when they went to the aid of the person who got electrocuted and is still touching the live wire...

Just as an side: I am a trained fire marshal and first aider. A few years back I was involved in a casualty simulation as described above. The simulation was used to show how we needed to think first, act second, and not add to the body count, and that in some situations, a body count is inevitable.

I hope you are now the wiser.

20
0

Re: This brings me happy memories

*Boot note: Oxygen suppression and Oxygen replacement is the same thing: Terms aren't exactly correct but I didn't reach for the manual before typing this. Bad me.

Heavy inert gasses displace oxygen, hence 'suppressing' or replacing it.

0
0
Mushroom

Re: This brings me happy memories

Worked at a major chemical plant on the NE coast of England, fire alarms everyone out, external sirens everyone in and close every exit tight. fire alarms and external sirens... find cupboard, stick head between legs, kiss ass goodbye.

7
0

Re: The point of Halon

ook !

3
0
Flame

Re: This brings me happy memories

Ahh... HALON, the IT's equivalent of a neutron bomb..

gets rid of pesky people leaving data centre unscathed. ^_^

3
0
Silver badge
Boffin

Re: This brings me happy memories

Let's prevent deaths from smoke inhalation by killing anybody who might be in the room wanting to breathe.

Only in the parallel BOFH universe. Or you are confusing Halon with CO2 flood.

In the real world Halon is almost inert and is not used in great enough concentrations to asphyxiate. If you were in the computer room when it dumped you would be off work with a monster headache the rest of the day and the next. It's not clear whether that's because of the huge noise it makes (only just short of eardrum-breaking), because of the flying ceiling tiles that the release usually causes, or because Halons have another use. The Anaesthetic gas that hospitals use is a halon.

Maybe someone once got fully anaesthatized, and that started the urban legend about it killing fires and people and leaving equipment unscathed?

1
0
Silver badge
Boffin

Re: This brings me happy memories

Halon is NOT an oxygen suppressant. That's CO2 flood.

Halon removes heat, not oxygen, in a rather clever way. When it is heated to its decomposition point, that is an endothermic chemical reaction. It absorbs heat, thereby cooling the flame. Further, the Bromine and other halogenated fragments that are released bond to the free radicals in the flame, so the flame is not only cooled but rendered incapable of catalyzing decomposition of its flammable substrate to generate more fuel for itself to feed on. It goes out and stays out.

It's exactly the same property that makes it so dangerous in the upper atmosphere, where UV decomposes it and where the fragments then catalyze the destruction of Ozone. Which is why it's been banned except in the most critical of safety-critical fire-suppression roles. Computers gave to take their chances with sprinklers, or be put in a true lights-out room with a CO2-flood system that kills both fires and people.

8
0
Thumb Up

Re: This brings me happy memories

@Nigel

You learn something new every day, and usually thanks to good folk like yourself willing to share knowledge. Cheers for an interesting read! :)

1
0
Anonymous Coward

Re: This brings me happy memories

I used to work there too, and worked on a project where I would call the on site ambulance service to say I am about start now, am wearing my breathing apparatus and have my gas alarms on, if the alarms go off its my body you need to carry out of the building when the all clear is sounded.

0
0
Bronze badge
Joke

Re: This brings me happy memories

I never got the point of halon.

Let's prevent deaths from smoke inhalation by killing anybody who might be in the room wanting to breathe.

One quick call to your PFY-supplier and you're back in business once the halon's cleared and the body dragged out back behind the smoking nut.

Dead PFYs and service engineers are less of a headache these days as you can always get a few "interns" to work for free...

0
0
Headmaster

Re: This brings me happy memories

Quote: Water and electricity do not mix

I was taught that they mix quite well. In fact they mix so well, that you should never try to be in the water while mixing it with electricity...

Just me olde 2 cents,

Guus

4
0
Bronze badge
Coat

Re: The point of Halon

Exactly. Halon (and its delivery) aren't things to monkey around with!

0
0
Trollface

Re: This brings me happy memories

Why not call the system an "oxygen displacement" system, since, as you point out, that's what they do. Ah, bueaucrats, can't live with 'em, can't suppress 'em.

1
0
Coffee/keyboard

Excellent

Eerily similar to an issue we had this week.. minus the Halon

Good Laugh!

0
0

Damn, that brought back some memories

In another life I was that PFY, in exactly that position. So much so I'm starting to wonder if my colleague's name back then was Simon Travaglia.

0
0
Facepalm

Top work as usual, though I feel it's a sad indictment of us BOFH wannabies when you have to bootnote a Halon reference

0
0
Bronze badge
Holmes

Halon Rocket Fuel

One place I worked was being developed and the spare Haylon devices were in the builder's store.

The builder's store took fire and one of the Haylon globes warmed up, and up, and up. In the end the outlet seal ruptured and, Werner Von Braun eat your heart out, the rocket was launched.

It went through several walls on its way to stardom, though not to the stars. The police investigator was not amused when he was told that the flying cylinder was an extinguisher.

A severe sense of humour failing followed

10
0
Thumb Up

Re: Halon Rocket Fuel

That is priceless.

1
0
Silver badge
Pint

Re: Halon Rocket Fuel

I've heard a similar story. Except it was a high-pressure air cylinder that someone had unstrapped from the wall and propped against another. It toppled, struck its neck on the bottom rung of the decorator's step-ladder, fractured, and went through three walls ending up embedded six feet into a hillside. No-one was killed. The decorator's deafness was temporary.

Or so I was told, over a beer.

(The REALLY scary stories involve overheated acetylene cylinders, which make unexploded WW2 bombs look friendly.)

1
0
Bronze badge

Re: Halon Rocket Fuel

I have a similar tale of an "F" size (about 4 foot tall and 8 inches wide) Medical Oxygen cylinder which was dropped in the gases store at an Ambulance station and snapped the valve off.

It went through (in order) a brick wall, three ambulance vehicles, another brick wall, the stone retaining wall at the side of the driveway and disappeared into a wood. Later found half a mile away...

You can still see the new brickwork where they patched up the building.

0
0
Holmes

Halon inspires striptease

Richard's rocket fuel story reminded me of something that happened in about 83/84. The then new halon fire suppression system was in two electrical loops, some under the floor, some in the ceiling void. Fairly standard layout for a big data shop (mainframes filled the rooms then - it really was big iron).

The system tests itself by having a trickle charge run through the electrical loop. This way it detects a break in the wiring. When you need to fire the halon, the deal is this: The siren goes and everyone heads to the exits; the halon is locked and can't go off until the shift supervisor has counted everyone out, locked the door, and activated the fire system. Simples.

Unless the control box has a hissy fit and decides to blow a fuse. Seems that in this case it assumes there is a fire, that the door is locked, and blows the circuit.

Pity the poor bod who was sat on a roll about chair in the bridge, under a large ceiling tile, when the globe above it fired. He got knocked off his chair and broke his arm.

Double pity the young female tape operator who was walking over an air grating when the globe under that went off. She had her dress ripped off by the blast and got out wearing only bra and knickers with an attitude to kill at 50'.

Cue another sense of humour failure!

14
0
Coat

Re: Halon inspires striptease

I think the most important question in response to that story is was she attractive?

I was going to go with 'hot' or 'a bombshell' but I think they are open to misinterpretation...

3
1

Page:

This topic is closed for new posts.