back to article BOFH: Do I smell burning toes, I mean burning toast?

BOFH logo – telephone with devil's horns There's trouble in the state of Mission Control. It has come to the attention of the Director that the PFY and I occupy a slice of priceless real estate complete with six full-height pivoting windows opening out to one of the few views which doesn't have a vast expanse of another …

Silver badge

Re: decisions HAVE been made!

I feel your pain, except for us our nice big server room became a massive meeting room (New false floor over the old false floor for added laziness), the server room is now in what was our store room (Which because we lost our other store room this week has now become both a server and store room which I can't keep flammable materials in >_<).

12
0
TRT
Silver badge

Re: decisions HAVE been made!

Sounds like an over temperature event is due any moment...

11
0
Silver badge

Re: decisions HAVE been made!

"Now the server room is 1/4 it's original size"

I take it Finance and Facilities are now having to share a server due to the reduced space available. A Raspberry Pi with a USB hard drive makes quite a useful server for combined spaces.

8
0
Silver badge
Meh

Re: decisions HAVE been made! - we had this happen

Our already modest server room was chopped in half and the 'extra' space taken to make a big conference room, including the removal of the raised floor and large Liebert aircon in that area. Then as time progressed and space was an issue, the 'conference' room had very small cubes installed to make a mini cube farm to accommodate the growth spurt we were undergoing.

2
0
Silver badge
Pint

not a drill?

In memory of Fats Domino"

14
0
Silver badge

On a clients site I almost fell out of a second story building window (double fire door) that could be opened fully to allow a loader to push a pallet into a server room. I still don't know why it had a push bar on it.

7
0
Silver badge

Entertainment value?

30
0
Silver badge

For me having to fill in the paper work for someone spilling tea on the floor is bad enough.

Anything like that would happen here it would be endless arm flailing and tick box forms.

2
0
Silver badge

At one site I worked at, the server room (on the third floor) had a number of rows of racks set at 90degrees to the external wall, and between each row was a large high window in the wall which was hinged at the top.

I was working in the rack nearest the wall, and leant on the window whilst trying to persuade a server into the sliders in the rack.

The window was unlatched, and opened under my weight, so there I was hanging out of the window with a large server in my arms, desperately trying to lever myself back in through force of will and a toe hold under the rack...

Interesting moment... :)

25
0
MJI
Silver badge

So you found where the real BOFH works.

21
0
Silver badge

"I still don't know why it had a push bar on it."

Did you notice the systems administrator was called Simon and had a side-kick with an acne problem?

8
0
Joke

"Anything like that would happen here it would be endless arm flailing..."

No, the arm flailing would end as he hit the ground :D

2
0
Silver badge
Windows

I keep wondering

Fishing line, open windows and a nicely charged fire pole.

I'll have to add that to the list.

(What? No, the prerequisite list for my next employer)

7
0
Silver badge

Re: I keep wondering

Pro-tip: If you grease the pole people end up going so fast when they get to the bottom they make a nice 'landing pile' for when you need to jump.

0
0

Not servers, but a museum that I worked in had the collections in a basement storage room. Great protection in case of tornadoes, which are common to the area. Except for the cast iron drain line suspended over the collections, ensuring that if there were a significant hit from a storm then all of the storm water would end up on the most valuable artifacts. They also kept the archival collection on the seventh floor, surrounded by glass windows. In an area prone to tornadoes...

13
0
Bronze badge

Ah - museums

One of the great problems of the hoarder mind set - and all decent museums are run by hoarders - is the inability to accept that there are limits on available space, and especially available space with appropriate climate control for fragile items (old texts, woven material, organic material, etc.). Dealing as they often do with bean-counting, penny pinching, space grabbing types, decisions are frequently made that resort to make shift lodging for critical collections. Occasionally the decision makers did not bother to inform the curators of the decision to relocate (or even OF the relocation). The school I went stored some collections in a structure know fondly as the "rat house" thanks to the large population of a hybrid wild/lab rat mix. Same school, after it was determined that the rat house needed to be razed for ?, the collection vanished to be relocated months later under the music building in an area contaminated by PCBs from the transformer. The engineer screamed aloud when he saw that "someone" had piled cardboard boxes full of flammable materials next to the transformer. My professor was unhappy as well and told the engineer that as soon as we could negotiate new storage space with university, the materials would certainly be moved. While negotiations were going on the collection once more vanished and was rediscovered several miles away at "the aquatic center" where the rowing team kept their shells and oars. Before it could be rescued, a winter storm came through removing the roof, dowsing the collection, doing massive damage to original paper work, and requiring hazmat operations - mold don't y'know - to rescue what could be rescued. Since the collection actually belonged to the US government we were able to point to the school administration and explain, "they did it!"

At another major university, the museum, renamed from a prominent anthropologist to a cranky 19th century, very wealthy woman who bequested an endowment to the school, relocated a large part of the collection to a space under the women's pool. The area was constantly exposed to chlorine gas. The consequences for the collection when constantly exposed to chlorine gas were unhappy.

10
0
Bronze badge

Re: Ah - museums

A couple of years ago the physicists at my uni got a brand new laboratory. The old building established in 1875 was taken over by the psychologists. Only then the huge cash of mercury under the floorboards was discovered.

4
0
Silver badge
Headmaster

Re: Ah - museums

Cache

Universities, eh?

2
0
Anonymous Coward

About 6 months...

after our building was turned over to the users and they were finally moving in (bulk of the build and fit out was bloody PFI with a French company that subcontracted to a load of different whoever-was-the-cheapest European builders during a UEFA Cup Final year which ensured the French plasterers, for example, "accidentally" emptied bags of dust into the open end of the vertical soil pipe the Spanish plumbers were currently fitting), a six inch water main blew on the main road and filled the basement to about 4 foot deep, luckily the animal house had only empty cages in at that point as the resident rodents etc hadn't been moved in yet. The HPC cluster was still installed down there a couple of years later, but the set of "lessons learned" watertight doors (manufactured by Canute Industries) and the concrete flood deflector (by Maginot Ltd) should keep the £mmm installation and all the rest of the main comms, servers and border links safe. Ish.

18
0
Silver badge

Kzzzrrrt!

Brilliant, but I still miss that old cattle prod.

5
0

"Could that ever really happen?"

"I could pretty much guarantee it," I say pointedly.

Brilliant!

8
0

At the previous company we were forced to move the office out of one factory into the adjacent one after a flood (fortunately the raised floor in the computer room meant nothing was damaged).

After pointing out that trying to cram the computer room and two people into a one-person office would be totally impractical, and possibly even illegal (insufficient volume per person) we were offered a semi-redundant toilet as an alternative. As part of the refurb we insisted that a dividing wall was removed, only to be told, "We can't, it's supporting a 1,000 gallon water tank!" Said tank turned out to be sitting on wooden beams between two walls and could easily have come down at any moment. We got it moved!

We also then discovered that there was an internal valley gutter running over the other end of the new room right above the comms rack. It only overflowed if the rain was really heavy... A year or so later and it was eventually fixed and we could remove the polythene sheet.

7
0

Ref the horror stories - Even when it all works...

One of our DCs was built to withstand everything, including no Grid forever (assuming diesel supplies). Six massive gennies in the basement (2 in, 2 spare, 2 servicing), a stack of UPS batteries that would have given Musk a chubby, and so on. Bloody impressive stuff, tested religiously each year. In isolation. Because as long as the components work, the whole does, right?

Turns out - eventually - we weren't testing all the components, because some of them had been hidden behind partition walls over the years, and disappeared off the documentation. Old building, been in use for a long time.

Then, one night, Local Substation went pop. I wasn't there, being sales, but I did get hauled over many coals by clients for weeks afterwards.

The sequence of events was, roughly, this;

1) Power goes

2) UPS - lifed at hour, tested a month previously, goes 100% to 21% almost instantly. Panic

3) Genny control panel does nothing (turns out it had it's own little genny we didn't know about, hidden away behind a stud wall). Main Gennies do nothing at all. More panic.

4) Someone finds the manual start button. Less panic.

5) Except Gennies won't sync onto the ring because the control panel is down. UPS down to 3%. Major flap now.

6) Read technical manual. Gennies can be manually synced onto the ring. "Sync them manually chaps, charge!" Except they can't. Every single breaker in 300,000sq-ft building tripped.

7) Brief pause of disbelief

8) One other, minor, technical thing then rears its head - the dead Genny control panel also controls the exhaust flaps. Massive generators still merrily chugging away with closed hall exhausts. Generator hall is now mostly not oxygen. FM200 system triggered. Everybody out.

9) For reasons I was never told, the water suppression on floors 4 and 6 also triggered... making floors 3 and 5 a bit damp too.

It was almost too farcical to behold - indeed, one of my banks actually said it couldn't have been that much of a cluster****. No Mr Bank, it was.

The supreme irony?

We were a DR provider...

32
0
Silver badge

The BIG red switch

Thing is who, when charged with "testing the backup power", actually goes to find the BIG red switch that disconnects the whole building from the incoming grid supply?

Its the only way to be sure. So if you want to really know, hire an Igor and ask them nicely to pull the switch.

Yes, the third switch!

15
0
Anonymous Coward

Re: Ref the horror stories - Even when it all works...

Interesting. I once worked for an electric utility. This meant that we knew just how reliable the mains power was. We therefore had a really big stack of batteries on our two UPSes, any one of which could have run the building (except for the process coolers in the DC) for four hours. We also had a diesel generator which could run the building, including the process coolers, and was fueled for a week. We tested the UPSes every quarter, by literally going to the main breaker and cutting off mains power. They worked. We tested the diesel twice a year, again by cutting off mains and seeing how long the automatic switch-over system took to start the generator without anyone having to go out back and start it by hand. The system was supposed to cut over to the diesel inside of two minutes, or long enough that it wouldn't start if there was a short power blip but fast enough that the temperature in the DC didn't rise noticeably. Every ever so often we'd top up the diesel fuel. So one day... the diesel spluttered and died. Panic ensued; we got the mains back on as fast as possible. It turns out that some smart boy had mixed kerosene into the diesel fuel, kerosene being a lot cheaper than diesel fuel, before selling it to the company. The generator didn't like it. We had to drain _all_ of the fuel from the generator (do you know how much fuel you need to keep a generator sufficient to run a DC for a week? Yes, that much.) and clean the gunk out of the generator, and we had to replace the burned out fuel pumps. The smart boy was sued flat; his defence was that he'd heard that diesels could run on kerosene. Yes, they can, but only if you prepare for it, such as by having the correct fuel pumps. Kerosene burns hotter than diesel oil, and has a lower viscosity, and will kill pumps. And rings. And pistons. And various other moving parts. If the company had known that we were getting kerosene, we could have made adjustments. Instead, the smart boy gave us a diesel/kerosene mix and charged us for straight diesel. Not only did he lose the civil suit, he got done for fraud.

27
0
Anonymous Coward

Re: Ref the horror stories - Even when it all works...

SunGard?

0
0

Maybe you're about have an epiliptic seizure.

https://canadianheritage.org/the-burnt-toast-canadian-heritage-moment/

3
0

explain/lie

May I present liesplaining as the new term for explaining with creativity?

8
0
Silver badge

Re: explain/lie

Approved.

0
0
Silver badge

The PFYs doctoral is coming along

Seems the PFY is getting ready to finally finish his doctoral level BOFH training. How long before Simon can safely retire in the knowledge the young PFY has been given all the knowledge he needs? (Or before Stephen attempts another coup)

4
0

"You’ll be given a chance to give your input"

I always take this to mean;

"The deadline to weigh in on this issue is in 5 minutes, and I'm not going to tell you that, so listen to the soothing sound of my voice and please take no action for another few days".

5
0

vengeful auto mechanic

Great story! Reminds me if a friend who showed his boss not to mess with his betters.

Picture, boss goes to open door to personal car, to go home for the day,the handle, it's the horizontal flip type, now packed full of Axel Grease. Boss notices rag conveniently left on ground, so wipes hands, and door handle, being a slob, throws rag back on ground. Gets into car, grabs ANOTHER Grease Filled handle on the inside. Steam rising off said boss, gets out of car, uses the same rag to clean up AGAIN, and said rag has pretty well had it by now, so, boss gets BACK into car, starts car, and grabs GREASE filled steering wheel, finally looses it, now pounding on steering wheel, near tears!!!

Meanwhile, said employee watches the whole thing covertly (-;

8
0
Silver badge

Re: vengeful auto mechanic

"Axel Grease"

Was that the well known Swedish disco star?

6
0

Server rooms

A while back I worked in a UK law enforcement department. They decided on a new building, as they did not want me to share their new office space, they told me I could have a desk in the server room. When they realised that I had an office (server room with restricted personnel access) bigger than an Assistant Chief, that was on the top floor with a view, and the only air-conditioned office in the building, the union forced the installation of £250,000 worth of aircon retrofitted to the rest of the building! This is where your tax money goes guys!

3
0
K
Silver badge
Trollface

I worked at a place just like this, must of had my desk shifted at least 8 or 9 times over the course... unfortunately we didn't have an underground garage though.

1
0

Equus africanus asinus

And the Director covets it in much the same way as the Boss covets his neighbour's ass – something I really should warn HR about.

And what is the name of this donkey the boss covets? Enquiring minds and all that.

1
0
Anonymous Coward

Bog off

We had a single pan toilet room with a small ante room converted into an office. All part of the no top down re-organisation presumably.

0
0

computers in the basement

A place I worked had -- going back to the old IBM 360 days -- the computers in B1 -- first floor just below street level. At the time they apparently thought computers were waterproof. Flash forward several decades, and the servers are much smaller, so the idea is, in case the river -- which was only a quarter mile east of us -- ever overflowed, when the flooding started it'd be all hands on deck, and we'd roll the servers into the elevator and bring them up to the 4th floor where they'd be safe and dry.

And then, someone realized, just in passing, mind you, that the motors for the elevator were in B2.

Finding a way to get the servers up the stairs was briefly considered.

1
0
Anonymous Coward

Shitty new build DC

I used to work for a mainframe systems supplier and commissioned a new £1 million mainframe in its brand spanking new £ 1 Million pound DC inside a 60 year old town hall basement, Partway through the install process I had a long process running and investigated a curious cupboard in the wall. I joked to my PFY that it probably contained a waste sewer not really believing anyone was so stupid. I actually expected to find an empty void but sure enough there was a freshly painted cast iron 6 inch waste pipe. (well the 10 foot of pipe within the dc was freshly painted).

I did query with the customer why they hadn't rerouted the waste pipe away from the DC and was told it hadn't been done as it would have cost £10,000. I pointed out the false economy and that I would not be available to help with Disaster Recovery when the pipe burst within the next 5 years.

1
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2017