back to article Sysadmin bloodied by icicle that overheated airport data centre

Why look at that: the calendar says “Friday” which means it's time for another edition of On-Call, The Register's weekly column in which we relate readers' recollections of being called out to fix nasty things under nastier circumstances. This week, meet “Alan” who “used to work for a rather large airport company in England …

Page:

  1. Anonymous Coward
    Anonymous Coward

    Welcome to the UK

    It takes a miracle of biblical proportions to make the core drill guy responsible for the install drill at an angle so it self-drains.

    In my own home office: "See these are the install instructions - you have to drill at 5 degree angle to ensure that water drains correctly out of the condenser". 2h later, the retard proud, gleaming and explaining with extreme pleasure about how he used the spirit level on his drill to drill it exactly level.

    I just sighed, paid him up and used a smaller pipe and some filler so it goes at an angle.

    1. pleb

      Re: Welcome to the UK

      I experienced the exact same mode of failure. Same exact solution, blowtorch and ladder. Except it was the domestic central heating condensing boiler condensate outflow. Installed by a very 'cartesian' gas fitter.

    2. Nick Ryan Silver badge

      Re: Welcome to the UK

      I worked in an out of town office where the numpty that installed the liquid run off pipe (soakaway) from the septic tank was carefully angled up hill. Apparently the installer didn't understand that liquids tend not to flow uphill. The solution was that we had to have the tank emptied regularly otherwise the surface of the car park became rather less than pleasant.

      None of this was helped by equally numpty cleaners who, despite being told repeatedly, continued to pour inappropriate cleaning chemicals down the drains.

      1. Symon Silver badge
        Coat

        Re: Welcome to the UK

        @Nick.

        Is this in the UK? The council in England where I live have something called 'building inspectors'. These folks make sure that everything meets the building control regulations. They take a special interest in small scale sewage treatment building works. There's no way a system with an uphill outflow could get signed off where I live.

        Also, compared to a cesspit, which is sealed and is designed to be emptied on a regular basis, a septic tank is small. It would take a matter of a few days to fill up. So, your office would have been emptying it once a week. The shit sucking people would have dobbed you in to the council, for sure.

        Hence, I'm very septical of your story.

        1. Dabooka Silver badge

          Re: Welcome to the UK

          Our system has an uphill outlet but it is pumped to the top of the drainage area which is downward. If that pump trips or the float switch gets caught (which happened after initial install until the cable was shortened and rerouted) you'd get a day or two max before the alarm would trip.

          It would make it impossible to live with if the run off wasn't working, the frequency of the tanker would be thrice weekly and cost a fecking fortune

        2. Nick Ryan Silver badge

          Re: Welcome to the UK

          (apart from my typos), I'm afraid that it's a genuine story of an office in Hertfordsire.

          The tank was definitely not small and the run off was continuous but as the car park was at angle so when the damn thing did overflow it did at least run off the carpark. Don't forget that as the tank was set underground, the overflow that should have been a last resort was somewhat higher than the top of the tank and therefore much of the runoff seeped straight back down into the ground, it's when the ground was too soaked or otherwise impermeable that problems happened.

          The tank was emptied pretty much every two weeks. I beleve that the shit sucking people were paid per visit and they were happy enough (well, as happy as you can be working in such things). The land was probably owned by a member of the council anyway...

          1. Symon Silver badge
            Big Brother

            Re: Welcome to the UK

            "The land was probably owned by a member of the council anyway"

            Say no more, now it all makes sense.

          2. Doctor Syntax Silver badge

            Re: Welcome to the UK

            I'll see your cess-pits and raise you a pig-farm slurry pit.

            There were a series of reports to the RUC that a couple had been killed and buried, the theme being that the bodies were dug up and moved. Nobody really believed it but it couldn't be entirely dismissed. (Have I ever mentioned the golf ball episode?)

            Anyway, one of the variations was that the bodies had been dumped in a slurry pit in a pig farm. This had to be pumped out so that it could be examined. Even empty it would be done by the underwater team with dry suits and breathing apparatus.

            The pit had been used for dumping carcasses of dead pigs and the pump was quite capable of drawing up and discharging bones. Somebody who was considered able to tell the difference between a pig bone and a human bone but cheaper than a regular pathologist had to spend a couple of days of an Irish winter standing at the outlet of the pump checking what came out. That was me. I suppose I should have been concerned about where the effluent was draining to. I wasn't

            The subsequent examination was negative and, I heard, very brief. I didn't see that for myself. I'd already left.

            1. herman Silver badge

              Re: Welcome to the UK

              In Penticton Canada, prostitutes from Vancouver were processed at a pig farm and mixed in with the pork sausages...

              I kid you not: https://en.wikipedia.org/wiki/Robert_Pickton

              The other jail birds subsequently removed him from the gene pool.

        3. Anonymous Coward
          Anonymous Coward

          a septic tank is small

          Never heard the term cesspit used in the US. We call the large, buried concrete tanks that are emptied periodically "septic tanks".

          1. Symon Silver badge
            Coat

            Re: a septic tank is small

            @Etatdame

            Apparently, in the U.S., cesspits are called holding tanks.

            https://en.wikipedia.org/wiki/Cesspit#Holding_tank

            A septic talk has an outlet for fluids. Depending on how careful you are with what you flush down the bog, a septic tank needs 'shit sucking' from every few months, to > 20 years. If Auntie Ruby comes to visit, make sure the jam rags go in a bin.

          2. Anonymous Coward
            Anonymous Coward

            Re: a septic tank is small

            Never heard the term cesspit used in the US. We call the large, buried concrete tanks that are emptied periodically "septic tanks".

            Ah, I stand corrected. I thought you called it Congress.

            :)

            1. Graham Dawson

              Re: a septic tank is small

              You can tell which is congress. Nobody with a brain would put a septic tank at the top of a hill.

            2. sanmigueelbeer Silver badge
              Pint

              Re: a septic tank is small

              Ah, I stand corrected. I thought you called it Congress

              *cough*Donald*cough*Trump*cough*,*cough*

            3. Morrie Wyatt
              Trollface

              Re: a septic tank is small

              Or alternatively, the "USPTO".

              Or maybe not. After all, a cesspit is designed to hold the shit, not issue it (with its mandatory collection of pluralities and overly broad vague hand waving about generic ideas) for the patent lawyers and NPEs to feed upon.

          3. ggray

            Re: a septic tank is small

            I know the term 'cesspool' which is probably the same in practice and meaning as cesspit.

            We were renting a house (rural community), built but not finished but the price was right.

            It was beginning winter but there was still some of the original summer garden growing

            which was a gift, given that it was over 50 miles to a decent store with veggies.

            There appeared to be a piece of plywood on some of the land (without plants) and I thought

            to move it and see if the soil could be worked and seeded with peas for the spring.

            What I saw was....undigested (not enough bacterial activity to work on waste) sewage. There had been enough activity to keep it warm (hence the growing plants).

            I realized where the seeds for the plants had come from and became sick immediately.

            We had not been told anything about this situation and the owner said we should have asked.

            He had hoped to sell the property to us. The place stayed vacant until he got a proper permit

            from the county and installed a septic system.

            What that entailed in a rural area was a large tank which would contain the waste, digest it, and the fluid which rose to the top would flow into a leach field where it would percolate into the surrounding area.

            The house was 40 miles from the county seat (NOT the area with the good stores) so the owner/builder had thought he'd get by doing things himself and sell it before someone found out.

            In California there is the 'grandfather' rule which is that what was acceptable at the time of construction would continue to be acceptable until alterations or additions were made.

            Consequently, those in the outlying areas knowing that building inspectors did not leave the county seat looking for code violations (often off the grid and on poorly maintained forest service roads) so that if violations were found, the owner would claim that it was grandfathered in. Used plumbing & electrical parts were in demand, often sold at flea markets or scavenged from the local dump.

            And the cesspools/cesspits? Since there had been more than adequate land available when the original houses had been built (acres), after a period of time, another pit would be dug, the pipes rerouted, the pit topped off with stones, wood waste, etc. and a tree planted in the middle of it.

            If you saw a magnificent tree where other plantings weren't so large or healthy, you knew what was there before.

            Even way back then (well before our time there) people had understood the grey-water/brown-water

            split so a leach field for grey water was only noted so the soil wasn't disturbed.

            The Irish have an expression about 'bad cess to you' (ill-wishing someone). Makes me think they might have used the term 'cess' for a pit or pool as well.

            1. Andy Taylor

              Re: a septic tank is small

              "What I saw was....undigested (not enough bacterial activity to work on waste) sewage. There had been enough activity to keep it warm (hence the growing plants)."

              Google "sewage treatment works tomatoes" :D

          4. Jim_E
            Happy

            Re: a septic tank is small

            In Australia rhyming slang for Americans is "septic tanks"

            Yanks -> septic tanks or seppos for short.

        4. Anonymous Coward
          Anonymous Coward

          Re: Welcome to the UK

          "The council in England where I live have something called 'building inspectors'. These folks make sure that everything meets the building control regulations."

          Really? In 2017?

          In most parts of the UK that I'm aware of, "building regulations" inspection has been outsourced, often to a subsidiary (or friend) of the builders. Bonfire of the red tape and all that. After all, what could possibly go wrong.

          Recently constructed PFI-financed Edinburgh and nearby Scottish schools found out what could go wrong a few months ago, fortunately without real casualties:

          http://www.edinburgh.gov.uk/news/article/2245/independent_report_into_school_closures_published

          "An independent report into the closure of 17 Edinburgh schools has been published today (9 February 2017).

          The School Closures Report (PDF 3.8MB) was commissioned by Chief Executive Andrew Kerr last summer after the collapse of a wall at Oxgangs Primary School and subsequent closure of 17 schools.

          It was led by respected construction and procurement industry expert, Professor John Cole CBE, who will present his report at a special City of Edinburgh Council meeting today.

          [snip]

          Some of the key findings from the report, which runs to over 250 pages, include:

          * the collapse of the wall was due to poor construction and inadequate supervision;

          * insufficient independent quality assurance and poor record keeping by the Council and ESP;

          * ineffective quality assurance measures within the construction industry;

          * the Council made the correct decision to close the 17 schools;

          * the alternative education arrangements put in place for over 8,300 pupils was a ‘remarkable feat’;

          the issues identified in Edinburgh are likely to be more widespread"

          The report contains many recommendations for the Council and other bodies, both public and private, as well as the construction industry. These relate to areas such as: procurement, construction, training and recruitment, the role of the building standards and independent certifiers and the sharing of information.

          [report continues...]

          "

          More recently and very much more tragically, there's little need for an inquiry to determine the root causes of what happened at Grenfell Tower; they're already well known to anyone who is interested, and some of them are even documented. The particular sequence of unfortunate events that led to Grenfell Tower isn't yet documented; the root causes that didn't stop the collapse in standards from happening are well understood already.

          We just need the relevant people, who are generally very well paid because they are supposedly in very responsible positions, to actually be held to account.

        5. DJ Smiley

          Re: Welcome to the UK

          LOL.

          Trust me, not all councils planning and buildings apartments are as good as yours. Here we've got amazon building a warehouse for 24/7 working, while promising us that the trucks won't have reversing alarms (a fact that'll only be in enforcement on our site, and these alarms generally can't be turned off for obvious reasons), that the overflow that occurs every time it rains, but never occurred before the building work started, is _nothing_ to do with the building of said warehouse, and the fact it's at least 3 small round about's away from any major road meaning the silent lorries are going to have extreme fun navigating around this little village (many small windy roads with 1400's architecture - so 6 foot thick sandstone walls and such - they will win against lorries ;))

          Also, septic tanks can be setup with soakaways depending on the amount of waste they're expecting and the proximity to things such as rivers and boreholes.

        6. TeeCee Gold badge

          Re: Welcome to the UK

          ...something called 'building inspectors'....

          Or "blind, incompetent idiots" as everyone else calls them.

          You want to really fuck something up? Get the quality control done by public sector staff.

      2. This post has been deleted by its author

      3. zb

        Re: Welcome to the UK

        Not just the UK, in my brand new apartment in Brazil all three showers had drainage points in the highest part of the floor. Must be something to do with being in the southern hemisphere and what Brazilians thinks of as down is really up

        They did it on the balcony floor too!.

      4. Alan Brown Silver badge

        Re: Welcome to the UK

        "None of this was helped by equally numpty cleaners who, despite being told repeatedly, continued to pour inappropriate cleaning chemicals down the drains."

        They tend to stop when given a choice of continuing to ignore instructions or being given a P45 (in the case of 3rd party cleaners, just threaten to cancel the contract for non-performance)

    3. fnusnu

      Re: Welcome to the UK

      You shouldn't have paid him: "Failure to meet customer quality expectations"

    4. Antron Argaiv Silver badge
      Thumb Up

      Re: Welcome to the UK

      ...you have to drill at 5 degree angle...

      I was betting it would be 5 degrees *the wrong way*...

      In my teens, when I was working on my first car, the shop manual offered this piece of sage advice:

      "Before putting all your weight behind a wrench, stop for a second and consider what would happen if the wrench should slip."

      I might not have been smart enough to apply this knowledge to the unfreezing of a pipe, but who knows?

      And a quick story from the warmer climes, where ice is not a problem: The wife's sister lives in Florida. Her aircon drain backed up, soaking her wood floor. Her husband investigated and removed a (deceased) large lizard from the drain pipe.

    5. Goopy

      Re: Welcome to the UK

      Or a pump.

    6. Anonymous Coward
      Anonymous Coward

      Re: Welcome to the UK

      "The facilities team quickly arrived, started lifting the data centre floor and quickly struck water “Literally everywhere underneath the floor.”"

      You didn't have leak detection equipment installed?! Never seen a datacentre without that....

    7. Alan Brown Silver badge

      Re: Welcome to the UK

      If the job isn't as specified then the dolt doesn't get paid AND gets to wear the cost of making good.

      Hitting fucktards in the wallet is about the only way to educate them.

  2. Anonymous South African Coward Silver badge

    Never had any snow or the such, but with the onset of winter our server room temperature monitor would spam us regularly with SMSes when the temperature dropped too low.

    Adjusting the aircon helped a bit (a few degrees upwards) but only for a time.

    Simply by adjusting the minimum level on the sensor did we manage to stop the SMS spam.

    Interesting to note that it only started to bombard us with SMSes between 18:00 and 20:00 - other times of the day it was all peachy and happy and quiet.

    1. Anonymous Coward
      Anonymous Coward

      Our office block in Pretoria was in a street of contiguous shops etc. Consequently it was tall and narrow - with offices on the back and front - which were respectively East/West facing.

      With sealed windows the temperature was maintained by air conditioning - cloudy days were rare. Unfortunately there was only one thermostat for our floor - and that was on the back of the building. So our office at the front had its temperature adjusted in inverse proportion to the sun shining through our windows. While the boss's back office had a constant pleasant environment - we either froze or fried depending on the time of day.

    2. Doctor Syntax Silver badge

      "Interesting to note that it only started to bombard us with SMSes between 18:00 and 20:00 - other times of the day it was all peachy and happy and quiet."

      Systems have a malevolent intelligence of their own. The AI folks are looking in the wrong direction.

      1. Arthur the cat Silver badge

        Systems have a malevolent intelligence of their own.

        It's known as Resistentialism.

  3. Anonymous Coward
    Anonymous Coward

    Years ago we had trouble with the temperature in the server room; the main air-con unit had failed and the backup unit couldn't cope and pissed water down the wall. When the service guys arrived, they found the external unit was no longer on the roof.

    It turned out the contractors relaying the roof had hacked through the pipes and cables to the unit and thrown it away - because they'd been told we didn't have any air-con in the building!

    The bean counters wouldn't stump up for a replacement, so for the next year or so we had to make do with the backup unit which dripped continuously into a large refuse bin that had to be dragged out and emptied every few days.

    1. DougS Silver badge

      Wouldn't they have been tipped off by the fact the AC was running or even if not the fact the electrical wires they were hacking through were LIVE?

      Not sure why they'd want to do all the extra work of hauling off an old AC unit even if it was no longer being used. Sure, it is easier to relay the roof without it in the way, but there's no way the time savings is made up for by the cost of removing a roof unit.

      1. Anonymous Coward
        Anonymous Coward

        Why would they do the extra work?

        You have a higher opinion of builders than I do. When I read "thrown it away" I mentally translated it to "installed it on another job and charged for supplying it" without any effort at all.

    2. Doctor Syntax Silver badge

      "The bean counters wouldn't stump up for a replacement, so for the next year or so we had to make do with the backup unit which dripped continuously into a large refuse bin that had to be dragged out and emptied every few days."

      Just have frequent precautionary shutdowns of any systems the beancounters rely on. After all, it's not good for the business to have no backup - which you don't if the intended backup has become the primary.

      1. I ain't Spartacus Gold badge
        Devil

        Or, just empty the bucket of water out the window as the beancounters are on their way out to lunch...

        1. Anonymous Coward
          Anonymous Coward

          Or, just empty the bucket of water out the window as the beancounters are on their way out to lunch...

          .. every. single. day. I'm sure they will get the message before the week is out :).

          1. kain preacher Silver badge

            Only if you piss and crap in it.

    3. Anonymous Coward
      Anonymous Coward

      Condensate

      I would have just put a condensate pump on the drain line and routed it out a window to drip continuously somewhere appropriate.

      It would save on dealing with the swamp bucket at least.

      1. Doctor Syntax Silver badge
        Thumb Up

        Re: Condensate

        "routed it out a window to drip continuously somewhere appropriate."

        The beancounters' window sill. Nice one.

      2. Wensleydale Cheese Silver badge
        Unhappy

        Re: Condensate

        "I would have just put a condensate pump on the drain line and routed it out a window to drip continuously somewhere appropriate."

        UK building regulations for domestic properties insist that the overflow from a cold water cistern in the loft comes out somewhere you can't ignore it, like above the front door.

        When the builders have skimped on lagging the cistern, said cistern freezes in winter, rendering the float valve inoperable. If you are unlucky, the water overflows through the ceiling. If you are lucky, the overflow pipe does its job.

        Ooh look, sheet ice just outside your front door.

  4. jake Silver badge

    Nice story, bro.

    First of all, he had to climb a ladder outside. So the data center's raised floor was above grade. And yet it didn't leak all over the lower level, but rather filled up like a bath? It's difficult to do that intentionally, accidentally is just about impossible. And how on earth could a long plug of ice have formed in a 10mm copper pipe in such a short period of time? If it was THAT cold, thawing it from the outside would have no affect internally. But it wasn't that cold, the data center was 50C, so no long plug was possible. And even if it were possible, the space under a raised floor is nowhere near enough to raise any kind of pressure, so it wouldn't have knocked him off the ladder.

    All's well with the story, though, because the facilities guys allowed a data center guy to ascend a ladder with a blow torch ... and we all know that that's going to happen when all kinds of things freeze over.

    1. Anonymous Coward
      Anonymous Coward

      Re: Nice story, bro.

      Agreed. This story is bullshit. Climbing ladders in winter carrying a blowtorche to melt ice in a small copper pipe that is causing an entire DC aircon to fail. Bull. Shit.

    2. Anonymous Coward
      Anonymous Coward

      Well, actually …

      We have a brand-spanking-new data center, and its floor is indeed designed to fill up like a bathtub so that, should the water cooling spring a leak, the waste water goes where it's supposed to, not to the floor below.

      Said data center has so far been severely degraded by a backhoe cutting a fiber that was less redundant than it was believed to be, and put off the air, first by a failure of cooling water supply from a central heating/cooling plant, then by a failure of the incoming 1MVA electricity supply and no fewer than three back-ups. Lucky a) that this is academia, and b) that I'm not responsible for it.

      1. Anonymous Coward
        Anonymous Coward

        Re: Well, actually …

        Having worked in an office with a raised floor, and having removed every single floor tile in the place when we moved in to do some rewiring, I can see how this could happen.

        The floor was several feet worth of reinforced concrete that didn't have holes drilled through it anywhere for fire safety, so that's not going to leak downwards. There is (obviously) no reason for holes drilled sideways through the building to the central lift shafts/stairs, and there wouldn't be in any case due to fire safety. The single centralised duct between floors was basically in a closet adjacent to the liftshaft, and this wasn't a bit that had a raised floor so cables to it went over the ceiling.

        So in my experience, filling up the cavity between the reinforced concrete and the floor tiles is possible, if you had an immensely huge source of water. That said, I wouldn't have thought that the AC units would have been producing *that* amount of water in a data centre, you only have so much humidity that can be sucked out of the air! (unless they were taking external air in at a high rate?) and once you had more than an inch then the underfloor power rails would get submerged, short and the RCD's/breakers would blow so I suspect the flood wasn't quite at biblical levels. ;)

        1. JimC Silver badge

          Re: power rails would get submerged, short and the RCD's/breakers would blow

          To my astonishment when we had a major AC plumbing failure all the servers and power breakers were working just fine with the under floor sockets and plugs under 6 inches of water.

          1. Anonymous Coward
            Anonymous Coward

            Re: power rails would get submerged, short and the RCD's/breakers would blow

            To my astonishment when we had a major AC plumbing failure all the servers and power breakers were working just fine with the under floor sockets and plugs under 6 inches of water.

            Without an RCD that can happen. It may even heat the water for you :).

            That said, the clock is ticking, don't expect it to last (and keep well away).

            1. JimC Silver badge

              Re: the clock is ticking, don't expect it to last

              Funnily enough as soon as we found the problem we threw everyone off the systems, shut everything down cleanly, turned the power off and got an AC engineer and a water pump.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019