back to article Sysadmin sticks finger in pipe, saves data centre from flood

Thank Crom it's Friday! At the end of today you can get on with what is best in life, which we hope includes reading this instalment of On-Call, our weekly look at legendary tales of IT gone wrong. This week, meet “George” who tells us he “once worked for a fairly large web hosting company which had a small data centre in the …

  1. Anonymous Coward
    Anonymous Coward

    Miaow

    “THE DATACENTRE IS ALL DARK EVERYTHING'S OFF IT'S QUIET AND IT SMELLS FUNNY AND I DON'T KNOW WHAT TO DO.”

    Go back to sleep, Ripley!

    1. The March Hare

      Re: Miaow

      Medlab - get to Medlab!

    2. Anonymous Coward
      Anonymous Coward

      Re: Miaow

      Somebody released the Halon and they're all dead.

      1. DiViDeD

        Re: Miaow

        Somebody released the Halon and they're all dead

        And yet still able to use the phones.

        Well, that explains our support desk.

  2. Richard Jones 1
    Stop

    The Power of Power

    Some years ago in a somewhat far away land a switch location needed to be redecorated. So preparations were made and the work of re-painting started. This was in the days of the metal paint pot. Bus bars tended to be protected on the sides and lower surface, but often had little or no such protection on the top which was close to the ceiling anyway. In fact there was enough clearance for a full paint pot plus a bit of clearance. So the painter placed his paint pot on the nice shelf he found, the pretty two colour one with a space between the colours.

    The predictable result was a high energy discharge of paint and a thrown can, plus a high velocity painter. The result was a catastrophic power loss. Crystal ovens lost power and burning paint added to the 'fun'.

    It took days to get even some of the site working again, the crystals sat in their irregularly shut down locations were as reluctant to stabilise as their down stream components were to work. Synchronisation of the set up moved from essential to would be nice through to rebuild and start again. Slowly flaky services were restored on an almost no redundancy basis until new parts could be shipped in and brought into service. I never heard what happened to the painter but his place in some Olympic running team was though likely if anyone could ever catch up with him.

    1. Anonymous Coward
      Anonymous Coward

      Re: The Power of Power

      You dodged a dead painter bullet there! Did Elf & Savetea update its manual, then?

      the crystals sat in their irregularly shut down locations

      Pardon me, but what is that?

      1. Richard Jones 1

        Re: The Power of Power

        The crystal ovens were part of the then solution to high stability crystal controlled oscillators for the internal synchronised digital networks, except when they were upset, which was very easy.

    2. Alister

      Re: The Power of Power

      the crystals sat in their irregularly shut down locations were as reluctant to stabilise

      Err, this is the Dilithium crystals, is it?

      1. Jim 43

        Re: The Power of Power

        I'm guessing oven controlled crystal oscillators. They drive the radios transmitters that play Jefferson Starship

    3. Adrian Tawse

      Re: The Power of Power

      I remember some years ago, when the VAX reigned supreme, there was a firm, actually still is, that had a data centre in London. This was supposed to be an ultra hi-rel data centre. It was supplied from two electrical substations, dueled up. Each VAX was dueled up and each had dual power supplies, together with a UPS each. What could possibly go wrong? Well, in the best of wiring best practices all this was controlled from a single switch room, which suffered a fire. All went black.

  3. magickmark
    Pirate

    CROM

    "Thank Crom it's Friday! At the end of today you can get on with what is best in life,"

    I plan to crush the weekend under my sandeled feet with no lamentation! And hopefully no electrical problems!!

    1. Simon Sharwood, Reg APAC Editor (Written by Reg staff)

      Re: CROM

      I was hoping someone would get that reference. Well played, Sir.

      1. tonelander

        Re: CROM

        The kids still have no idea what the regular Dad-joke meal-time summons is from.

        "Dinner for wolves ... on the table now".

      2. magickmark
        Mushroom

        Re: CROM

        Wow - my first reply from a genuine bonafide El-Reg employee!

        I think I need to go lie down for a while!!

      3. Midnight

        Re: CROM

        I was reminded of this guy instead...

        "Friday at last!" crowed Conan springing from his mats with the agility of an antelope. "Crom's Beard but it took long enough to get here!"

        -- Conan the Salaryman

    2. phuzz Silver badge
      IT Angle

      Re: CROM

      Thank Cron surely?

      17 01 * * 5 phuzz mv phuzz pub

      (I'm sure someone can come up with a better crontab entry than that)

    3. Oliver Reed

      Re: CROM

      And by Hanumans balls too!

      1. Triggerfish

        Re: CROM

        When they ask you what an achievement is in a job interview, apparently "Crush your enemies. See them driven before you. Hear the lamentations of their women." is not the correct response. Which I have to say completly goes against my impression of the average boardroom.

        1. dmacleo

          Re: CROM

          OMG that hit my funny bone hard :)

    4. Anonymous South African Coward Bronze badge

      Re: CROM

      Take a pull on the hellhorn for me when you get to Gehenna first!

  4. Sgt_Oddball

    There's nothing quite like...

    The smell of a freshly toasted UPS. It's really fun too if it starts emitting smoke while it's at it... That was a fun afternoon shutting down non essential kit sooner than anticipated because a number of servers lost one of the two UPS hooked in.

    1. Chewi
      Unhappy

      Re: There's nothing quite like...

      I have a (relatively small) UPS in my living room. I'm quite scared now.

      1. Sgt_Oddball

        Re: There's nothing quite like...

        Small ones ain't so bad, these were ones with 2 car sized batteries in apiece.

        We'd have two hooked up into each server (though 2 PSU's for redundancy) but after that incidence it got changed to scrapping the older ones, keeping the newer ones and buying one mega UPS that had about 18 car sized batteries in it's own cage away from everything else. The bigger one was also by it's self was good enough to keep the server room up for 3 hours.

        We then also put solar panels on the roof and directly fed that in too so there was a supplementary charge should the worse happen.

        1. Yet Another Anonymous coward Silver badge

          I love the smell of burning UPS in the morning

          It smells like overtime.

        2. TeeCee Gold badge
          Alert

          Re: There's nothing quite like...

          ...these were ones with 2 car sized batteries in apiece....

          Best "UPS" lashup I ever heard of was in Turkey. A roomful of secondhand car batteries, of varying sizes capacities and vintages, with an old transformer of indeterminate origin across them, all mounted on something that looked like Dexion racking fabricated from scrap angle iron.

          It worked, but the fumes in there meant you had to come out for air every ten minutes should you be unlucky enough to have to work on any of the other kit in there.

          They don't really do "elf 'n savdee" in Turkey. I also saw some lads fitting airco units to an office building. Two problems. First was that the bloke fitting the exchangers was hanging out of a window with two other blokes holding his belt. Second was that he was securing the exchanger brackets to the marble cladding panels on the outside of the building. He had to pick where to put the brackets, as many of said cladding panels had already fallen off, despite not having a heavy heat exchanger bolted to them. I mentally flagged the pavement beneath as a "beware of falling heavy machinery" zone.

      2. Anonymous Coward
        Anonymous Coward

        Re: There's nothing quite like...

        You should be scared, one Uni I know well has had a spate of them failing, often with toxic smoke, and more than once with big flames, and that's just the desk top ones. Poor battery connections are a big part of most UPS failures, along with swollen batteries when it causes overheating, at that point the Hydrogen evolved becomes an issue, with the overheating connection causing ignition. Still whats a £175,000,000 claim against not paying for a professional to install a 300KVA UPS, one of their bigger f'ck UPS.

    2. Anonymous Coward
      Anonymous Coward

      Re: There's nothing quite like...

      before we had a new DC built with a UPS supplying it we had rack mounted UPS, must have had around a dozen. Anyway one day a horrid smell in the server room (just after the AC had been serviced) anyway blame went towards the AC engineer and the AC. But he swore it wasn't anything to do with that, trouble is with a false floor and bottom of rack mounted UPS its very difficult to pinpoint where the smell is coming from and all the UPS looked fine. That is until we took the plastic front off and touched one of them and found it to be bleedy hot! Shut it down got it out of the rack and put it out in the yard, stinking!

      1. Chris King

        Re: There's nothing quite like...

        An old tale I've told before, but worth retelling...

        AC starts leaking in a LAN room, and a rack-mount UPS takes a shower, shorting out and catching fire in the process - giving us the unique situation of a LAN room that's flooded and on fire at the same time.

        My boss asked how this was possible, and was told "Well, we usually have the fire on top and the flood below !"

        1. Stoneshop

          Re: There's nothing quite like...

          AC starts leaking in a LAN room,

          Well, a computer room, not a dedicated LAN room, although it involved the LAN. And it wasn't much of a computer room either, merely a repurposed office where they had installed a raised floor to house a pair of VAX8200's.

          The problem was the LAN getting gradually flakier, so I was called in to figure out what was happening, and solve it. After lifting a floor tile to check where the AUI cable went, it immediately became clear what the problem was: apparently the aircon had sprung a leak, and there was a serious quantity of a strange yellowish fluid under the raised floor. Pulling on the cable it transpired that the coolant had dissolved the linoleum flooring into something custard-like, engulfing a DELNI and some other unidentifiable stuff.

          Getting a few AUI cables and a new DELNI, simply put on top of one of the systems, took care of the LAN problems; whatever they did with the custard was NMP.

    3. Anonymous Coward
      Anonymous Coward

      Re: There's nothing quite like...

      Yeah, I had the switchboard at a fortune 500 company go down when one of the three inverters on the UPS blew on a Saturday night, causing the single breaker to which all three inverters had been connected to trip.

      The UPS gamely kept the switch running almost all Sunday, but the service provider had not only failed in the electrical installation but had failed to set up that particular alarm, so nobody knew. A quick flip of the bypass switch on Monday morning calmed the receptionists, but the service provider lost that particular maintenance contract.

  5. Aqua Marina

    Speaking of boarding up things.

    A long time ago I had a satellite office down in Dudley, that had several servers located at the side of the main office. Usual story of no budget for anything, and this was before the days of remote admin and email. I had macgyvered together a system that monitored temperatures inside the cases, and if the office got too hot an alarm would sound, and I would be called in to intervene, usually by telling them to turn the heating off and open a window (the girls would turn off the AC and turn on the heating, rather than say wear something longer than a belt of a skirt, or a cardigan over the strappy sleeveless top). This system worked quite well.

    One day I get a call that everything had stopped working. I rush the 100 mile or so journey down, and find 2 dead servers, and my alarm wailing away. The server room was like an oven. Hold on.. room? what room, where did this room come from. I queried one of the staff, who told me that some months beeping was coming from the servers. The office manager didn't like this noise, so he arranged for a studded wall to be built around them, and a lockable door.

    For months this alarm had been going off, and the office manager had simply been ignoring it when he dutifully went in each morning to swap over the backup tapes because "why would the room being too warm damage anything?"

    1. SImon Hobson Bronze badge

      Re: Speaking of boarding up things.

      the girls would turn off the AC and turn on the heating, rather than say wear something longer than a belt of a skirt, or a cardigan over the strappy sleeveless top

      Ah, you've had that too.

      The ones who turn up when there's snow on the ground wearing some very summery outfit and complain that they are cold - at the same time that some of the women "of a certain age" are complaining that they are hot. Any suggestion that they wear something more fitting for the conditions is met in the same sort of way as if you'd been proposing a dirty weekend away with them.

      Of course, you're expected to keep the office warm enough, and cool enough, and all without any sound or moving air.

      To top it off, the way to the canteen was through the factory - and although they tried to keep the walkway clear, a natural consequence of what the factory did meant that it was always a little on the slippery side. Again, suggestions that the office girls wear some more sensible shoes was also met with "a certain level of disdain".

      1. Michael H.F. Wilkinson Silver badge

        Re: Speaking of boarding up things.

        We are now well into the fourth day of a heatwave (30+ Celsius), with girls sitting in the sweltering sunshine outside of an air-conditioned cafetaria, wearing thick black jeans and woollen jumpers complaining it is too hot. It seems some people decide what to wear by looking at their wardrobe (and various fashion magazines), and then expect the weather to accommodation their whims. Worse, some think I am weird coming to work in t-shirt, shorts and sandals (and, yes, I am weird in wearing just a t-shirt and jeans in midwinter, but you will never hear me complain about the cold) .

        1. G.Y.

          humidity Re: Speaking of boarding up things.

          At one place, half the people complained it's too hot, half complained it's too cold. The AC crew ran around with a thermometer, but had no humidity meter; I suggested the thermometer be replaced by an astrology book -- no less accurate (for purpose) and more entertaining.

          But one employee would ride her bicycle to work, hang up her wet sweatshirt -- and it was still wet 9 hours later.

      2. Hollerithevo

        Re: Speaking of boarding up things.

        Women are idiots and the old ones should be put out of their misery, amirite?

  6. Anonymous Coward
    Anonymous Coward

    Smell

    The Selenium rectifiers we had in an old medical X-ray machine we were using for NDT work smelled like boiled cabbages when they blew.

    1. Anonymous Coward
      Anonymous Coward

      Re: Smell

      Thanks - I was about to eat breakfast and you remind me about the smell of selenium rectifiers... My stomach is churning even though it was 30+ years since I last smelt one.

      1. Michael Strorm Silver badge

        Re: Smell

        Not surprising. Selenium is immediately below sulphur in the same column of the periodic table and apparently even worse in terms of smell.

        In fact, it gets two mentions in the "Things I Won't Work With" chemistry blog, here and here.

        The element below *that*, Tellurium is apparently worse still(!), so bad that even half a microgram absorbed into the body can cause your breath to stink of garlic for over a day. Greater quantities can give you the same effect for weeks(!!)

        Nice!

    2. Anonymous Coward
      Anonymous Coward

      Re: Smell

      Great thing about selenium and burning epoxy is that they both smell toxic enough that you tend to avoid the fumes. The reason cyanide and carbon monoxide are so dangerous is not the lethal dose - it's lower for sulphides and selenides - but that nobody can smell CO and many people cannot smell HCN.

      1. Will Godfrey Silver badge
        Happy

        Re: Smell

        Ah yes, the good old selenium rectifier. Once smelt, never forgotten. Most 'ordinary' people are quite amazed when you walk in a room and say "Hold on, I'll just nip out to the van and get another rectifier."

  7. wyatt

    I've had 3 power related experiences:

    1) Army, Kosovo, standby satellite link to the UK. Following battery maintenance I was told to replace the batteries into their tray in the vehicle and connect them up, whilst it was working. Not being the easiest of jobs when not powered on, there were sparks flying everywhere a few times and almost brown trousers.

    2) Army again, Field Trials of new equipment. Generator changes were a physical disconnect followed by connecting the cables to the next generator. UPS had to take the strain whilst this was happening which was great, if it had all the batteries connected up..

    3) Last night shift and the internet went down. Called my mate in London in our other office who also had no internet. Checked the comms room and it was very dark and quiet.. No Finger Printing ability for the police for a few days..

  8. Anonymous Coward
    Anonymous Coward

    Reminds me of the headquarters of a well known financial organisation. They had a duck pond in the grounds.

    I'm sure you can guess what happened to the comms room underneath it. I think the drip trays are still there.

    1. Anonymous Coward
      Anonymous Coward

      "I'm sure you can guess what happened to the comms room underneath it. I think the drip trays are still there."

      Ripley: Lieutenant, what do those pulse rifles fire?

      Lieutenant Gorman: 10 millimeter explosive tip caseless. Standard light armor piercing rounds. Why?

      Ripley: Well, look where your team is. They're right under the primary heat exchangers.

      Lieutenant Gorman: So?

      Ripley: So, if they fire their weapons in there, won't they rupture the cooling system?

      Burke: [interjecting] Ho, ho, ho. Yeah, she's absolutely right.

      Lieutenant Gorman: So? So what?

      Burke: Look, this whole station is basically a big fusion reactor, right? So you're talkin' about a thermonuclear explosion and adiós, muchachos.

      Lieutenant Gorman: Oh, great. Wonderful. Shit!

      (Anonymous, so I can't be directly accused of just copy/pasting a quote).

      1. Yet Another Anonymous coward Silver badge

        Followed by 6months of meetings and the decision to add a sign

        "Do not fire automatic weapons into the nuclear reactor"

        sign is then taken down because it doesn't meet corporate guidelines and doesn't use the official Weyland-Yutani font and doesn't have icons for the hard of reading

    2. Chris King

      At least that water would be reasonably fresh...

      ...but how about a LAN room situated underneath a gent's toilet ?

      With leaky urinals ?

      1. Anonymous Coward
        Anonymous Coward

        Re: At least that water would be reasonably fresh...

        our old server room (2nd floor) used to sit under a wet lab (3rd floor) several times the Millipore still in the lab would overflow and result in water coming through the ceiling of the server room. But at least it was VERY clean water!

        1. Rich 11

          Re: At least that water would be reasonably fresh...

          We used to have an AV lab located under a top floor of student residences. Guess how long brand-new student digs have to be occupied before someone leaves the bathroom or kitchen tap on overnight?

          Four days, apparently. Running water and state-of-the-art video-editing suites don't mix very well.

        2. John Brown (no body) Silver badge

          Re: At least that water would be reasonably fresh...

          "Millipore still in the lab would overflow...But at least it was VERY clean water!"

          So, basically, free additional cooling by an non-condictive liquid, yes? :-)

      2. This post has been deleted by its author

      3. Anonymous Coward
        Anonymous Coward

        Re: At least that water would be reasonably fresh...

        @Chris King; "A LAN room situated underneath a gent's toilet ? With leaky urinals ?"

        I'd say that was taking the... ah, you got there before me.

  9. Alan Brown Silver badge

    location

    "I'm sure you can guess what happened to the comms room underneath it."

    I can never fathom the mentality of any company which puts their essential electronics/electrical kit in the basement or under the water tanks.

    Anyone who encounters such a setup should specifically point out the risks and make sure it's passed to the beancounters. They tend to understand the actual costs of outages and will "make management understand" why XYZ is a bad idea.

    ALWAYS keep the beancounters onside. They are your best allies in any corporate structure.

    1. imanidiot Silver badge

      Re: location

      Beancounters are a weird quatum entity. They can be your worst enemy or your best ally. Oftentimes both at the same time. And you don't find out which until the quarterly budget reports collapse the waveform.

      1. WonkoTheSane
        Pint

        Re: location

        Schrödinger's beancounters?

        Pint, because not just Friday, but PAYDAY! :D

    2. collinsl Bronze badge
      Coat

      Re: location

      How about in a basement under the water tanks?

      Mine's the one with the dad joke book in the pocket

    3. Anonymous Coward
      Anonymous Coward

      Re: location

      me either! Our bean counter was very keen on building our new DC in a very large underused store room which would have been ok'ish apart from being no where near any services (power, lift, up some steps, small entrance, etc) and our preferred location was next to the existing server room with all the services you need and right next to the lift, etc. Oh and the one slight issue was it was also next to a CHUFFING great BIG refrigerated wet lab containing about 6 large seawater tanks. He seriously wanted this location so he could converted our preferred location in to meeting rooms, despite us saying no and every DC contractor saying ermmmm not a good idea! If it wasn't for the lab being in a conservation area and planning issues about siting the AC fan units he might have got his way!

    4. Martin an gof Silver badge

      Re: location

      I can never fathom the mentality of any company which puts their essential electronics/electrical kit in the basement or under the water tanks.

      I've searched in vain for the reference now, but I'm certain that when the National Archive opened their new building near the Thames, there was a news story about the fact that they had decided to put all their "infrastructure" - building systems, IT etc - in the basement, below the groundwater level and at risk of flooding, because as far as they were concerned it was sacrificial and nowhere near as important as their physical records which remained firmly above ground. Presumably they had an offsite DR plan for the IT, and who cares if you end up having to mop out a comms room, but trying to extract the Thames from 500 year old unique documents is nigh-on impossible.

      That said, at my current place of work, air conditioning units, heating pipework etc, directly above the racks. Who would have thought that a leak would even happen, much less cause problems? Just before I started here they had to deal with just such an event and as a result, the racks in each rack room now have "roofs", sloping to a bit of guttering and piped to... well... somewhere less vulnerable, and fitted with a leak detection system.

      M.

    5. Anonymous Coward
      Anonymous Coward

      Re: location

      See also Manchester Computing Centre circa 20-odd years ago, with Kilburn Building. A building designed in the sixties for a seventies-era supercomputer, but with odd, disturbing human touches.

      Things like the rooftop garden complete with pond that was situated DIRECTLY OVER the main computer room. Little things like that, which occasionally leaked.

      The pond has gone now, to be replaced with a skylight soon for a single, huge open-plan office for all IT Services staff (of whom there will be ever-fewer as people discover just how loathsome a huge open-plan office is, and depart for greener, saner pastures).

      1. Yet Another Anonymous coward Silver badge

        Re: location

        That was one of the stories from the hurricane that hit New York.

        Data center on an upper floor - check

        Backup generators on top floor - check

        Diesel tanks for backup generators on roof - check

        Pumps to supply more fuel up to rooftop tanks in lowest basement - .......

        1. swampdog

          Let's do a generator test..

          Let's do it at the weekend when no-one is there. Check.

          Lets not inform facilities. Check.

          Test begins. 30 minutes later, total outage.

          Post mortem.

          Management: The generators failed. Why?

          Facilities: They won't run on air. Bugger all diesel.

          1. imanidiot Silver badge

            Re: Let's do a generator test..

            @Swampdog, so you get informed of powerfailures requiring the generator so Facilities can fuel them with diesel? Kind of a strange explanation there, I would assume backup generators should be fueled and ready to go at any time.

            1. MrT

              Re: Let's do a generator test..

              "I would assume backup generators should be fueled and ready to go at any time."

              Have you never seen The Dish... ;-)

        2. OzBob

          Re: location

          Yep, had that one too, at 2pm of the afternoon before Y2K ticked over. Fortunately no major problems.

      2. Anonymous Coward
        Anonymous Coward

        Re: location

        Almost as good as Southampton's old data centre mainframe in the bottom of one of their towers, below the water table and the local stream that becomes a torrent after heavy rain, in what was a 'funk-pit' for the Uni's great and good to hide in should the cold war have gone hot, the bilge pumps power supply wasn't backed up... 4 feet of water certainly improved the things cooling, to stone cold dead.

  10. Anonymous Coward
    Anonymous Coward

    A few years ago . .

    I started a contract at a little company with a relatively isolated office. They had UPSes and a backup generator and on a regular cycle would test that the generator was operational etc.

    About six months before I arrived the power failed. The UPS took over, the generator powered up smoothly and took the load . . for about fifteen minutes until it ran out of fuel.

    1. Anonymous Coward
      Anonymous Coward

      Re: A few years ago . .

      Better still, a "technical republisher" I worked at forgot to put *any* diesel in the generator's tank. A tree fell in high winds, taking out the power lines to the building, which was in a fairly remote area. The generator made a single "cough" sound, belched some smoke and failed to start.

      1. SImon Hobson Bronze badge

        Re: A few years ago . .

        Ah, the old generator trick.

        Starts fine when there's mains, but doesn't when it needs to because the batteries aren't very good.

        Or the diesel is many years old and full of biological sludge - there's a bug that lives in diesel tanks.

        Or it's just simply clogged up from all those "start, run with no load, stop" cycles because there's no way to put it under load without a short power drop.

        Or, the diesel tank is in the basement and the fuel pump is mains powered.

        (in the UK) There are outfits you can sign up with who will manage your generator and get you STOR payments from the grid. It will need some switchgear changes for many customers (to allow the generator to parallel with the grid), but once that's done you can fire up the genny and run it at full load for a proper test. Further, when it gets called on for short term grid supply, you get paid for it and get to turn over your fuel - you get paid a small amount just for having it available.

        STOR = Short Term Operating Reserve. This is the "oh sh*t" reserves the grid can call on when (for example) a large generator goes offline without warning.

        1. This post has been deleted by its author

        2. Anonymous Coward
          Anonymous Coward

          Re: A few years ago . .

          Diesel bug, bio sludge, call it what you will, is a problem for most backup gennies that gets overlooked, there are additives that help, and chemical cleaners to kill the bacteria, but most don't use them.

          Simply changing the fuel every year or two won't cut it either, the slimy residue infects the new fuel and clogs fuel filters very quickly. I use fuelset additive and avoid bio-derv where ever possible in my land-rover, but it's not so easy with a 5000ltr genny tank to keep the fuel fresh and clean, unless you buy a fuel polishing pump and filter unit and use it regularly with fuelset or similar. Even if you can buy fresh red Dino diesel without added bio-diesel.

          Then there's the issue of the size of the genny, most places install the bare minimum, one set I know of is now less than 40% of the required capacity, its built into the building so tightly they can't get it out and nothing else would fit, planning permission for an external containerised genny has been sought, with lots of objections, then there's finding the money to buy it and installing it...

          1. Benno

            Re: A few years ago . .

            Just a few weeks ago, I had to provide 1kW of emergency power from my car inverter to a site that lost mains and someone was too slack to make sure that the generator was serviceable. I heard that it may have damaged the starter trying to get it to run with no fuel...

    2. Richard 26
      Facepalm

      Re: A few years ago . .

      It didn't happen to me personally but there is the story where a similar thing happened. The problem wasn't that the tank was out of fuel; it was that nobody had thought to put the fuel pump on a protected supply.

    3. Anonymous Coward
      Anonymous Coward

      Re: A few years ago . .

      Many years ago I recall this happening on a rather grander scale when the oil refinery I was working at lost grid power and the on-site gas turbine which could have kept it running failed due to a lack of diesel in its fuel tanks. Oops.

      It wasn't a lucky gas turbine that one. It was disabled by an internal explosion during startup on another occasion. Then there was a time when it was working perfectly during another grid outage only to trip after a switch failure meant it was trying to power half the country rather than just the refinery.

    4. anothercynic Silver badge

      Re: A few years ago . .

      Hmmm, yes... the genny. Everyone forgets that the genny needs *fuel* to run. :-)

      We had that once on our Wednesday DR switchovers once... Took our source control admin weeks to resolve that because the database backing the source control system up died. Cue the slow claps.

      1. rhydian

        Re: A few years ago . .

        Its not just Generator controls that sometimes get overlooked when distributing UPS feeds.

        You'd be surprised how much telecoms termination equipment (Especially fibre) isn't UPS fed...

  11. Anonymous South African Coward Bronze badge

    Last weekend I replaced the batteries for all the UPS'es in the server room. 3x RCT UPS'es and one APC UPS. All of them take 6x 7Ah batteries.

    The servers connected to the RCT UPS'es was shutted down before, and started up after battery replacement. No problems yet.

    The APC UPS I bunged into bypass, replaced the batteries, and bunged it back into line mode, also without issues. This was our COMMS UPS.

    So far so good, but I wouldve appreciated it even more to have a full CO2 fire extinguisher at the ready... still waiting for refills. Some d00fus have decided to use all of our CO2's for traning, and I'm still waiting for the CO2's to be refilled/replaced - have sent a strong letter off to manglement regarding this.

    Would love to see what happens should we have to use a powder-based extinguisher instead of a CO2 on server kit...

    1. Anonymous Coward
      Anonymous Coward

      Would love to see what happens should we have to use a powder-based extinguisher instead of a CO2 on server kit...

      "I have installed PowderShell on every server..."

    2. Oengus

      Another use for CO2 extinguishers

      Some d00fus have decided to use all of our CO2's for traning

      The doofuses at one location I worked used to use the CO2 extinguishers to cool their tinnies. When the inspectors came around no one knew why the extinguishers always needed recharging.

  12. SImon Hobson Bronze badge

    Best generator tale I've heard was this.

    At a previous employer, we had our own substation, and when they needed to do tree clearing work, the sent someone to our site to isolate our end and be sure that we couldn't backfeed the line. I got talking to the guy and he told me this tale ...

    They were doing some work on the lines ito a nearby town, and had brought in a couple of portable generators. These were hooked up and running, but under no load.

    A manager was walking past, in his clean hi-vis just as the load went onto the generators. As the power went on, anyone who knows about diesels will realise that all sorts of oily sh*t comes out the exhaust - and this manager was head to toe in black oily smuts from the diesel exhaust.

  13. Unicornpiss
    Pint

    Something like this?

    Electrical problems?

    Bit of a Bang..

    Equipment replacement plus one soiled office chair to replace for the guy sitting just outside of camera range..

    1. Anonymous Coward
      Anonymous Coward

      Re: Something like this?

      Bit of a Bang.

      Facehuggers in the circuitry?

      Why did nobody install the rotating red beacon to give atmosphere as required by evil company regulations?

      1. John Brown (no body) Silver badge

        Re: Something like this?

        "Why did nobody install the rotating red beacon to give atmosphere as required by evil company regulations?"

        Blame the CCTV installers. The flashing red beacon is just out of top right of shot. I was a bit more concerned that someone appeared at the left edge of frame just seconds before the gas went off.

  14. Anonymous Coward
    Facepalm

    Old story

    I do remember a story that did the rounds many moons ago. Apparently a computer facility somewhere in the US decided to test their backup power systems by physically shutting down the mains power. Everything started OK; power went off, UPS takes up the load, then the diesal generators kick in and take over from the UPS.

    Everything works absolutely swimmingly. For about 30 sceonds, after which point the generators stutter and die, followed shortly afterwards by the UPS which did not have the capacity to keep the facility going for long (they wrere only sized to handle the load until the generators spin up to full capacity). Queue "oh sh*t" moment.

    After a long-ish investigation the source of the problem was identified. Apparently the fuel tanks for the diesal generators had to be located some distance away for fire safety reasons. Because of the distance a gravity feed to the generators was out of the question, so they used fuel pumps instead. Which (you may have guessed it) where powered directly from the mains with no feed-in from the backup power systems. Hence the generators worked for until they have sucked the fuel lines dry, then shut down.

    Queue secid "oh sh*t" moment plus much forhead slapping.

  15. simonpearse

    One of my customers, a Taxi company, had a few PC's connected to a small UPS with a small 2-stroke generator as backup.

    The computers were wired to their own rig main connected to the ups. The generator was kept in a locked cupboard. There had been no mains issues for several years until a digger severed the mains supply in the street.

    Over the years everything and its brother had been connected to the 'UPS ring main' We're talking kettles, electric heaters etc, so the ups lasted a few mS before failing.

    They couldn't find the key for the generator cupboard. so after a while the door got forced. The petrol was years old and they had lost the starting instructions. After an hour of getting a sore arm someone noticed that the shop next door still had power so they went looking for a long extension lead. Unfortunately this would only get one PC going so some genius came up with the idea of removing the socket from the lead and replacing it with a plug. The plan being to plug said lead between the shop next door and a socket on the ups 'ring' so that all the sockets would be live. They reasoned that this would keep everything working, and once power was restored they could simply remove the extension lead.

    Those familiar with 3 phase may look away at this point.

    1. Anonymous South African Coward Bronze badge
      Mushroom

      Ooooh, don't get me started on 3-phase

      Couple of weeks ago power started to fluctuate real bad at my company. Got a sparky out from the shitty council, he said all power is OK and walked away. He only tested between the three phases, but not between each phase and neutral (for 230V)

      Cue a private sparky who was doing investigation for another company, commandeered said sparky to have a quick shufty at the power, and discover serious fluctuations between all three phases and neutral.

      Transformer eventually popped, and the shitty council replaced it with a brand new unit. BUT they wired up the phases wrongly, so 3 phase electrical motors ran in reverse. Said a few choice words and they came out again to fix their futtup.

      1. Will Godfrey Silver badge
        Facepalm

        Should have taken them to the cleaners. Just a couple of seconds of reverse drive on some printing presses can do hundreds of thousands pounds worth of damage - to say nothing of the cost of downtime.

        1. Unicornpiss

          reverse running..

          To say nothing of 3-phase refrigeration compressors!

  16. Bodge99

    Home UPS

    Back in the early days of decent sized LCD monitors I bought a few 27" ones for the family (and me).. 9 in total. A short while after this I was offered a "job load" of "industrial" UPS's. These contained hugh batteries and weighed a ton. Just perfect for a household full of desktops..

    Anyway, about 6 months later we had a power outage in middle of the day. Each UPS kicked in as required. Unfortunately the "pure sine wave" UPS output wasn't at all pure (more like a lumpy triangle wave when I 'scoped it). This resulted in smoke pouring out of each monitor after 30 seconds or so.

    Yep, 8 monitors that were powered at the time were destroyed.. bummer!

    Moral: If buying a "pure sine wave" UPS or generator, ask for proof that the output really is "pure".

    1. Callam McMillan

      Re: Home UPS

      Or buy a decent UPS! Personally I'll only go with APC Smart-UPS systems now.

    2. BinkyTheMagicPaperclip Silver badge

      Re: Home UPS

      Yep, been there, done that when work bought some cheap UPSes from a second hand IT kit place. The output wasn't pure enough so kit just refused to work on some UPSes, and only worked when we turned sensitivity right down on others. APC Smart UPS worked with no issues.

    3. swampdog

      Re: Home UPS

      My home UPS was killed by a couple of (for want of a better way to describe it) "bullet" power outages: On/Off many times a second. The UPS couldn't switch fast enough and if I was lucky the servers would power off. The unlucky servers would be in an even bigger mess.

      I never scoped my UPS. It was enough to plug a mains light bulb into it and see it flickering so much it was only just one shade brighter than yellow. I've never replaced it because we still get such outages. Plus the fact they scare me. I still have a scar from a 6v VW Beetle battery!

  17. Korev Silver badge

    This shed used to protect the JANET* POP for the East of England... http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/misc/shed/

    * UK academic network

  18. earl grey
    Unhappy

    It's all about the smell

    Years ago i worked in a computer facility that had some of the older Unisys mainframes in which the large capacitors were filled with what smelled like fish oil when they started going bad. At that point (if you were lucky), you could get the customer engineer involved to track down the misbehaving unit before it totally failed (and brought the system down).

  19. Anonymous Coward
    Anonymous Coward

    Cafeteria Flood

    One of my clients moved into a new building and put their data center, along with the payroll department, in the basement. It had raised floors everywhere except one room. You can guess where the data center was located, yep, in the only room without raised flooring. That room was also directly below the cafeteria so when the feed to the ice machine broke it flooded. We had two racks and a mini computer in a flooded room.

    Strangely, their management couldn't fathom how bad this was and continued to keep the (now replacement) computers in that room after we mopped and dried it.

    Nice.

  20. kstieers

    Ironic start to backup power issues...

    When I joined my current company in 2003 they'd been using APC units, one per rack in a data, and hoping they didn't lose mains. As the ERP project that was going to centralize much of the operations ramped up, and the number of racks went from 4 to 8, we were finally able to convince management that we needed a room-wide ups and a generator.

    The day the generator showed up on the flatbed trailer, we had a power outage. IS director and I gave the CFO a pointed look....

    A few weeks later the generator, UPS and transfer switch had been installed and it was time to test... We killed the mains, the UPS picked up the load, the generated started up, the transfer switch did its thing and the UPS saw the power, and took it...perfect... Then mains were turned back on, the transfer switch did its thing, a couple of minutes later the generator shut down, and the electricians left...

    I stayed late to do some patching or something, about 30 min later I lost access to some servers. I walked down to the data center to find a silent room... A panicked call to the electricians got the instructions to get UPS to take main power again.

    Turns out that transfer switch wasn't set to wait for the generator and mains power to be in sync before they switch back, so the UPS saw the switch to main power as a nasty spike...

    Thankfully the clean up wasn't bad...

  21. Herby

    First liar doesn't get a chance...

    My generator story: My parents house suffered a power outage along with some parts of the neighborhood. Turned out to be a pole transformer. Called the local utility who suggested to my dad "Why don't you take your wife to dinner" when just then a bunch more guests showed up doubling the mouths to feed. My task was to get the generator running (it was a small two cycle thing) and at least get some lights up. Eventually after fiddling with all the fuel and air controls on the generator I was able to get it going. Unfortunately it was only half of the 120-0-120 service. Being ever so clever and knowing that the range worked mainly on 240 volts, I engaged all the burners and the oven to get to the other half of the service. We did a bunch of barbecuing that evening, and I stuck a BUNCH of labels on the generator so I could figure it out "next time" (which really hasn't happened yet).

    Of course none of this really relates to IT, as home computers weren't common (it was the 70's).

    1. swampdog

      If we're sidling off-topic

      Many years ago I saw a generator in Aldi stores for either £129 or £229 (I forget). It just begged me to purchase it but my missus forbade me. Admittedly she did have the logical high-ground in that it would be useless to us but she has never been sat drinking beer in the back of a small boat..

      ..where the guy had constructed a home-made generator out of a Colt Suffolk mower engine, a fan-belt and an alternator, all mounted on a piece of wood. As we clamber aboard my mate catches the throttle and before you can say "f*ck me it's disappeared over the side", it had vibrated itself off the deck into the river.

      1. Unicornpiss

        insurance

        I bought a gasoline generator about 5 years ago when the weather was calling for "the snowstorm of the century", which of course never materialized. Mostly I just wanted to keep my hydronic heating system running to heat the house and prevent damage in case everything froze up for days and I lost power. And my neighborhood had suffered about 1 outage a year lasting an hour or more.

        Of course after purchasing the generator, the power hasn't gone out in the entire 5-year period since.

        1. Doctor Syntax Silver badge

          Re: insurance

          "Of course after purchasing the generator, the power hasn't gone out in the entire 5-year period since."

          Obviously a wise investment. Just think what the weather would have been like if you hadn't bought it.

        2. Anonymous Coward
          Anonymous Coward

          Re: insurance

          I hope you run it on a regular basis and keep fresh gasoline on hand anyway. Just cycle the old stuff through your car (or mower or....)

      2. Anonymous Coward
        Anonymous Coward

        Re: If we're sidling off-topic

        don't you just love Aldi! Best random item I've seen for sale. Wheelbarrow Wheels! thousands of em! I mean how often do you EVER change a wheel of a wheelbarrow!

        £9.99 if anyone is interested and I think they're back in stock!

  22. J. Cook Silver badge
    Pint

    Couple short ones here...

    Both are sort or second hand, though.

    1. my boss at $ISP where I used to work related this to me: the site he was at had a power outage, and the generator failed to start up- didn't even crank.. Fortunately, they had massive UPS units, so they had some time to run to the local auto parts shop and buy a new battery for the generator. Turns out one of the overnight NOC staff had a bad battery, and swapped his dude for the one the generator was using.

    2. Same company; One of the things I did working for $ISP was travel to their various NOCs doing field work on their routing and switch gear. I was at the Austin NOC, and got the tour by the site manager. I noticed that one of the big DC PDUs had a notch in the power plane and mentioned it to the manager. Apparently, an electrician was working on it using an uninsulated nutdriver and managed to short the power and ground plane (-48v, Many amps) blew the electrician to the wall 30 feet away from the PDU, put the nutdriver *through* the wall, and blew the upstream breaker.

    Said electrician survived, but was (understandably) banned from the site.

    I have a picture of the notch, too: http://sub-ether.dreamhosters.com/gallery/v/jecook/work_horrors/archive/AUSTIN002.jpg.html

    Fortunately(?) I don't have any power horror stories which I witnessed first hand.

  23. Mark 85

    Not my story but I was involved in a round about way. Back 1970ish.. out of the military and in college. I took a job as "the" maintenance man for a 20 story office building in the downtown area of my hometown. The building was built in the very early 1900's and had cast iron riser pipes (those being the pipes that pumped the supply water to the top floor where they then u-turned and went down to feed each floor water. The city changed it's water supply to "soft" water in the early 60's and so the pipes were lined with lime build up. In the two years I worked there, I walked into work a total of 6 times to find that one of the risers blew out when the lime inside had disappeared. I won't go into why management didn't replace them all after the first one blew. Oh.. it started life as a bank building so there was a vault in the basement next to my office in the basement.

    Fast forward.. I go to my high school's 40th reunion. One of my former classmates is CIO of a company that now owns the building. After having a great evening, I visited him on Monday and got to see my old office... etc. The old maintenance office in the basement was the NOC. They were using the sub-basement for the datacenter. Nice digs and I told him the tales of the pipes breaking and flooding to midway up the basement wall. We had a few laughs and I returned back to the west coast.

    More fast forward... you probably know where this is going. A few years later, I get a call.... "I remember you said you worked there. Facilities want's to know where the sump pumps are?". Me: "There used to be a steel door next to the vault and it's a stairway to the sub-sub basement where the sump pumps are located. Why?". The door had been covered during a remodel and everyone forgot about it, not that it would have helped. They had a riser break (one of the ones I had replaced many years ago) and flooded the sub-basement (DC) and the basement (NOC) and water was pouring out of the elevator shaft into the main lobby and then to street. I advised him to call the fire department and see if they could pump the water out after they hired a diver to go to the sub-basement and turn off the water supply.

    All I know now is that the company is still there and their DC and NOC are on the 5th floor. Yes, my classmate is still CIO. I'm going to my 50th reunion in about 3 months and hope to see him.

  24. anonengineer
    Alert

    TV studio gets indoor waterfall

    I used to work as a broadcast engineer at a *major* UK facility a few years ago.

    Long story short - we were in the middle of a multi-million pound rebuild, moving the majority of the studio control areas as well as rebuilding the server and transmission rooms from top to bottom. Elsewhere in the building, rooms were being refitted and walls were being moved - requiring large changes to the sprinkler system. The wonderful fire contractor sawed through the pipe on the second floor and rather than re-connect or plug it - left it and went to work on another job (you can see where I'm going).

    Another building at the studios conducted a fire drill, evacuating about 1,200 people from a 'fire', activating the fire alarm... which started the water pumps in-case of a deployment.

    The result? an entire water tower emptied into the TV studios at the tune of 1,200 PSI (it was sufficient to blow a hole in a concrete wall). Fortunately the contractor cut the pipe directly above the main server room.

    The result? almost every ceiling tile blew out on the ground floor emptying tons of water into the newly built server rooms and studio control areas. Amazingly though *most* of the expensive stuff hadn't been installed yet, so we just had to dry out the sound proofing and repair some kit. The IT bods took most of the water and, with no power in the building and no shows replaced the dead kit once the building was dry and the contractor replaced.

    Probably not my best day at the studios. Nor that contractors (having about 30 engineers out for his blood).

  25. Airstreamer

    So much power, and no way to use it..

    Yep, company I work for put a major data center in the basement of a building. I made a crack about bilge pumps and got a lot of dirty looks. Well, we got a about a foot or so of snow outside, and things were fine... Until we got a temp rise of about 40F in two hours. The vacant lots "upstream" of the vaults started melting, and guess where the water went? Yup, right in through those nice big 4" conduits that exit the foundation right above the DC power, the phone switch, network gear and other assorted goodies.

    Also, they installed not one but TWO 1.8 Megawatt (yes, megawatt,) Cat Diesels to run the building. Well, the whole shebang only needs about 100kw, but they forgot to set it up so that either diesel could supply the two 100kw UPS units.. Guess what's next? YES! POWER OUTAGE and only ONE diesel starts! One UPS finally dies, and half the place goes dark....

  26. Major_Variola

    FLIR cameras are not that expensive...

    Perhaps data centre attendants should have one to survey the gear...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like