back to article Heatwave shmeatwave: Brit IT departments cool their racks – explicit pics

Never let it be said that techies aren't agile or innovative – and perhaps a little slapdash – when solving problems on a tight budget. Only this week, El Reg was given early sight of a pre-patented swanky new cooling system. IT departments at businesses that asked to remain anonymous, for obvious reasons, have devised a …

  1. LDS Silver badge
    Flame

    I always like when people put flammable materials...

    ... close to equipments that may overheat or send sparks around...

    1. Crisp Silver badge
      Coat

      Re: I always like when people put flammable materials...

      The fools! They should use inflammable materials.

      1. StargateSg7 Bronze badge

        Re: I always like when people put flammable materials...

        "...The fools! They should use inflammable materials....."

        Careful..........Americans might read this and take your words to heart!

        They won't know what to do.........

        As a Canadian, I can appreciate the distinction, but just in case, I did notice some stainless steel tanks with an "Inflammable Liquid" logo sticker on it that I MAYBE suggest using as an IT gear coolant. Just spray it on the hottest parts.....and see if it cools down your gear to a great enough degree!

      2. tinman
        Mushroom

        Re: I always like when people put flammable materials...

        jest ye not, I know a senior engineer who thought that inflammable means doesn't burn...

    2. Hans Neeson-Bumpsadese Silver badge

      Re: I always like when people put flammable materials...

      That's why you always know where the nearest fire extinguisher is. For example, mine is currently acting as a door stop to keep the fire door open for ventilation purposes.

      1. Yet Another Anonymous coward Silver badge

        Re: I always like when people put flammable materials...

        That's why you always know where the nearest fire extinguisher is

        They are all currently locked away in the H&S dept because we don't have an official document specifying if the test date is month/day/year or day/month/year so the test label is invalid so the extinguisher can't be used.

        1. Oengus Silver badge

          Re: I always like when people put flammable materials...

          They are all currently locked away in the H&S dept because we don't have any staff certified to use them (and no one will put up budget to go on a training course).

      2. short a sandwich

        Re: I always like when people put flammable materials...

        The British Standard door stop. Also the bane of H&S people's lives at this time of the year.

        1. Yet Another Anonymous coward Silver badge

          Re: I always like when people put flammable materials...

          Couldn't we just take all the old extinguishers, empty them and use them as door stops?

          Environmentally friendly recycling, and since they couldn't be used as extinguishers H&S can't complain

          1. John Brown (no body) Silver badge
            Facepalm

            Re: I always like when people put flammable materials...

            "Environmentally friendly recycling, and since they couldn't be used as extinguishers H&S can't complain"

            They would probably ask for and get a budget increase approval to have them painted a special non-fire-extinguishy colour with a large notice on saying "NOT A FIRE EXTINGUISHER" and send all staff on a half day compulsory H&S refresher course to make sure everyone was aware that the new recycled door-stops are NOT FIRE EXTINGUISHERS, all at a far, far higher cost that just buying in a few quids worth of door stops.

            1. Yet Another Anonymous coward Silver badge

              Re: I always like when people put flammable materials...

              No it would just not have an up to date test label signed by the correct authority - absence of this renders it unable to be used and so there would be no confusion.

              We had to replace all the simple color coded red/black/cream/blue extinguishers with identical red ones and the type written in small letter on the label. So simply labelling these as contents "none" for use on "no fire types" would be sufficent and fully in line with H&S policy

          2. stefbishop

            Re: I always like when people put flammable materials...

            Yes hold the fire doors open!

          3. Mark 85 Silver badge

            Re: I always like when people put flammable materials...

            Couldn't we just take all the old extinguishers, empty them and use them as door stops?

            I know a guy who took a rather large CO2 extinguisher and converted it to a beer tap. Rather impressive to draw a beer and quench one's fiery thirst.

            1. The Oncoming Scorn Silver badge
              Pint

              Re: I always like when people put flammable materials...

              The same technique was used on illicitly brewed, store & dispense a quick shot of beer if you wanted, on at least one oil rig, until some H&S type decided to test it\real fire & found a different sort of foamy goodness coming out.

    3. Mike Lewis

      Re: I always like when people put flammable materials...

      When I started working at one company, I saw that they were storing half-empty paint tins in the same cupboard as the fuse box. That cupboard was located between our offices and the stairs, cutting off escape if it caught fire. I got my manager to move them.

  2. DonL

    Temperature ratings

    For these reasons, when purchasing network equiptment, I now take operating temperature ratings into account. Some equiptment can accept very high temperatures, making additional cooling during summer unnecessary.

    I never had to use a fan or had heat related outages, but I know these small patch racks can get quite hot at times.

    1. RogerT

      Re: Temperature ratings

      But the suits grumble about the extra 0.5% cost.

    2. big_D Silver badge

      Re: Temperature ratings

      One company I worked for had a rack in a south facing 3rd floor room, with large windows. The CEO got air con in his office, but he declared the IT had survived this long without AC, it didn't need it.

      The "trick" was, the first person in in the monring opened the windows wide to allow the air to circulate... :-S

      I installed a thermometer in the room and in the rack. Average summer temperature in the room was 38°C. The middle of the rack was approaching over 60°C!

      Interestingly, we only had one server throw a hissy fit, an 8 year old HP server. The rest (only 6 years old) all ran stably throughout the summer! We did however borrow an air compressor in the June and cleaned the dust out of every machine in the rack, 6 years worth of dust isn't good for the lungs!

      1. CrazyOldCatMan Silver badge

        Re: Temperature ratings

        6 years worth of dust isn't good for the lungs

        How about 10 years-worth of cat hair and associated dried mud? That's what I vacced out of an old server once (at home I hasten to add).

        These days, my home computer room door is kept shut[1][2] - prompted by one of the cats being sick all over the network switch. Which meant buying a new one since half the ports stopped working once the stomach acids had done their work on the circuit board..

        [1] Much to the annoyance of senior female cat - she regards any closed door as a personal affront to her dignity.

        [2] Just as well that many years ago, we had aircon fitted to that room - back in the days when I was a contractor and had my own limited company. Which paid for my then motorbike and the computer room aircon.

        1. Doctor Syntax Silver badge

          Re: Temperature ratings

          "Which meant buying a new one"

          Maybe the new one would have been equally sick.

        2. Yet Another Anonymous coward Silver badge

          Re: Temperature ratings

          one of the cats being sick all over the network switch.

          That's because you were only using Cat-1 cable.

          Cat5 can withstand upto 5 cats, Cat6 even more - obvious really

          1. Anonymous Coward
            Anonymous Coward

            Re: Temperature ratings

            "Cat5 can withstand upto 5 cats, Cat6 even more - obvious really"

            Indeed, it's the most resistant to feline vomit. That's why it's called Cat Sicks.

      2. Linker3000
        Mushroom

        Re: for the cost of a tape to copy them to and delivery.

        At one place I worked, the in-house facilities guys turned one end of an office block into a fully airconned computer room with raised flooring:

        1) They boxed-in a row of radiators behind drywall - but didn't shut them down, so from the getgo the room never reached the expected temperature. The aircon guys spent ages recalculating things, checking equipment etc., before someone commented 'does this wall seem warm to you..?' Out came the padsaw, holes were cut and valves were turned. The room temperature dropped, but the ugly holes were never fixed.

        2) They put the room stat on a pillar next to a window so it was affected by outside temperature and sunshine. The room went into superchill mode when the sun was shining, and on very cold days the aircon would hardly kick in and the room stayed toasty. When someone put 2+2 together, the stat was relocated.

    3. Alan Brown Silver badge

      Re: Temperature ratings

      "Some equipment can accept very high temperatures, making additional cooling during summer unnecessary."

      If you setup your patch rack correctly then any internal fans will be more than adequate to push airflow through and out of the cabinet - and if you think they're going to get hot then specify one with appropriate fans and acoustic baffling in the first place.

      One way of achieving this is to ensure that the first places to lose network connectivity in the event of overheating are the offices of the beancounters, sales and HR.

  3. MrKrotos

    Why is this even a story?

    This is normal for us Brits, I remember building a server with bits from Simply Computers many years ago. It had a ton of 9.2GB SCSI drives in it that ran hot enough to fry an egg on, obvs we took the front panel off and stuck a desk fan infront of it to blow through. Worked like a champ for years :P

    1. Chris King Silver badge
      Flame

      Re: Why is this even a story?

      Those sound like Micropolis disks. No need for a space heater in winter when you've got a couple of those in your workstation.

      That, or Quantum Fireballs. The most aptly-named disks I ever used because they always felt like they were going to spontaneously burst into flame after a few hours use.

      Icon, because toasty disks - AIIIIEEE !!! hot hot HOT HOT HOT !!!

      1. big_D Silver badge
        Thumb Up

        Re: Why is this even a story?

        I hat a few QFs. A mate had problems with SCSI termination and was going to through a SCSI card and 4 drives in the bin. I nabbed them off him, set one jumper and had a blindingly fast set up on my PC!

      2. CrazyOldCatMan Silver badge

        Re: Why is this even a story?

        No need for a space heater in winter when you've got a couple of those in your workstation

        Ditto for a Dell server with 8 15K RPM drives in the front. The acoustic case mostly muffled the noise but didn't do much for the heat generated. Still, it kept upstairs nice and warm in the winter..

      3. Alan Brown Silver badge

        Re: Why is this even a story?

        "Quantum Fireballs."

        the original 4GB Seagate Barracudas (long before Quantum started naming drives) _required_ forced ventilation and not having it meant loss of warranty. At the time at 7k rpm they were the fastest drives on the market and sounded intimidating as they spun up (they were also $3500 apiece)

  4. Hans Neeson-Bumpsadese Silver badge

    That last picture

    I applaud your use of a cable, rather than the more traditional gaffa tape, to secure the fan in position. If, however, you accomplished this job whilst standing on a stepladder rather than balancing precariously on an office chair, then you have lost all respect that I might have had for you.

    1. MonkeyBob
      Mushroom

      Re: That last picture

      I assume the cable was secured by plugging both ends into the same switch.

      1. Korev Silver badge
        Joke

        Re: That last picture

        Tying it to some wood also works well. I believe they call that a "Spanning Tree" in network circles

        1. Sir Runcible Spoon Silver badge
          Coat

          Re: That last picture

          Pin it in place with a toolbox and you've got yourself a Spanner Tree!

  5. Claverhouse Bronze badge
    Mushroom

    Wrong Type of Leaves

    There's an opportunity here for some bright sparks to invent some form of cooling computers.

    British businesses should ask how they do it in California.

    1. tfewster Silver badge
      Facepalm

      Re: Wrong Type of Leaves

      The issue is that we don't often get hot weather in the UK, so proper cooling would be a "waste of money". ISTR that UK Elf n Safety regulations specify the lowest temperature staff can be made to work in, but not an upper limit.

      Apparently the business being shut down by overheating kit heat isn't a problem?

      1. WonkoTheSane
        Headmaster

        Re: Wrong Type of Leaves

        Elfin Safety specify no absolute limit either way, but instead rely on the vagueness of the phrase "During working hours, the temperature in all workplaces inside buildings shall be reasonable."

        1. John Brown (no body) Silver badge

          Re: Wrong Type of Leaves

          "During working hours, the temperature in all workplaces inside buildings shall be reasonable."

          I take that to mean the old Shops, Offices and Railway Premises Act, which did have a minimum working temperature has been superseded then? (IIRC, the minimum temp. had to be reached within an hour of the start of the work day.)

    2. Sir Runcible Spoon Silver badge

      Re: Wrong Type of Leaves

      British businesses should ask how they do it in California.

      I believe they use this quaint thing called 'money'. IT Depts. in the UK don't get to see a great deal of that.

    3. StargateSg7 Bronze badge

      Re: Wrong Type of Leaves

      "....There's an opportunity here for some bright sparks to invent some form of cooling computers.

      British businesses should ask how they do it in California...."

      ---

      Here in Western Canada, especially parts in southern Alberta (i.e. a Western Canadian Province), some parts are a TRUE DESERT which have temperatures as high as 45 degrees Celcius (113 F) in the summer so Alberta oil company systems engineers who had servers in that area had an ingenious solution! Cutting Fluid!

      Quad CPU server motherboards (Tyan brand usually) and hard drives (Seagate ATA or Western Digital SCSI) were encased in large powder coated aluminum electrical mains boxes that you could buy at any major construction supplies supplier for less than $30 CAN each or about 20 Euros! The powder coating was used to coat the entire interior and exterior of the case so the cutting fluid wouldn't react with the aluminum and the electrical and network connectors, which were the ruggedized kind. They sank 20 such cases into a 1 metre cubed tub and ran the cables to the outside connections. Cutting fluid is usually used in metal machining to cool the milling bits as they cut through stainless and very hard cobalt steels.

      It's the perfect fluid to cool electronics! Just NOT in direct contact with the motherboard because the cutting fluid IS ionic and therefore CONDUCTIVE ... BUT...if you dip a PROPERLY SEALED aluminum mains case with a motherboard or drive into a pool of the stuff, the cooling power was amazing.

      The motherboards were running NO HOTTER than 40 to 45 Celcius at FULL LOAD because the heat transfer was so good even in the hottest part of summer! They just made sure the CPU cooler fins had their fins cut and fitted so the CPU heat transferred directly to the aluminum case and then out to the external cutting fluid.

      The entire process SHOULD have been patented, but the engineers (mostly oil and gas electrical and mechanical engineers) were just solving a local problem! Anyways, it's worked for DECADES! Some of those 2 and 4 core Quad CPU servers are STILL running after nearly 15 to 20 years untouched processing oil and gas reservoir drilling data! Even the DRIVES are still running after 15-to-20 years which is nearly UNHEARD OF in the industry! You would think after so many years the capacitors on the motherboards and the drive bearings would have dried out or broken down after being on 24/7/365 for 20 years!

      Actually it showed just how WELL Tyan mottherboards and Western Digital SCSI drives were built in those days! They weren't cheap! BUT they are STILL running!

    4. shedied

      Re: Wrong Type of Leaves

      In a rare display of candour, and ingenuity, their IT types will happily point out that their servers all faced Westward--that is, before the seasonal Santa Ana winds came and the hardware were engulfed in the blaze along with a few dedicated-but-hopelessly-outdated fire extinguishers

  6. Rich 11 Silver badge

    Have you hit upon an interesting way to cool your tech systems?

    If you have critical kit which regularly gets above 45C, tightly wrap it in 20 metres of 6mm clear plastic tubing. Insert one end of the tubing into one upper femoral artery of your youngest and healthiest intern and the other into the basilic vein of the opposite arm once all the air has been forced from the tubing. The intern will act as an active heat sink for your critical device.

    1. sabroni Silver badge
      Happy

      Re: Have you hit upon an interesting way to cool your tech systems?

      Do you vote conservative?

      1. CrazyOldCatMan Silver badge

        Re: Have you hit upon an interesting way to cool your tech systems?

        Do you vote conservative?

        Either that or is from the US..

        1. Rich 11 Silver badge

          Re: Have you hit upon an interesting way to cool your tech systems?

          Either that or is from the US..

          No. If I were from the US I would have used obscure variations on medieval units of measurement rather than bog-standard metric.

          (With apologies to the El Reg Standards Soviet for the obvious omissions.)

        2. Potemkine! Silver badge

          Re: Have you hit upon an interesting way to cool your tech systems?

          Either that or is from the US.

          If from the US he would use a migrant's child rather than an intern. There are plenty of them available for a cheap price.

          1. Rich 11 Silver badge

            Re: Have you hit upon an interesting way to cool your tech systems?

            Goddammit, I haven't even finished writing the UK patent yet and now you're telling me I have to rewrite it for the US.

            No rest for the wicked...

      2. Korev Silver badge
        Coat

        Re: Have you hit upon an interesting way to cool your tech systems?

        >Do you vote conservative?

        Sounds like he's found an artory

      3. Rich 11 Silver badge

        Re: Have you hit upon an interesting way to cool your tech systems?

        Do you vote conservative?

        Of course not! What sort of sick fuck do you take me for?

  7. Anonymous Coward
    Anonymous Coward

    Ceiling or floor?

    Always wonder why installers often place up near ceiling where temperature higher than floor and nowhere for warm air to rise.

    Guess it's to keep floor space free for desks. Even so, not rocket science thermally...

    1. Stoneshop Silver badge

      Re: Ceiling or floor?

      Those 10U wall racks aren't usually that big a problem regarding heat, though. Half the height is patch panels, one or two cable management panels, three or four switches at maybe 100W each. A decent rack design can deal with that.

      Mine (10U) takes 130W max, and internally it's about 5 degrees above ambient

  8. Anonymous Coward
    Anonymous Coward

    Once upon a time

    I, and half a dozen other people, worked in a stuffy windowless basement which was where the prototype hardware we were developing ROM code for resided.

    In vain we pleaded for aircon. The temeperatures were up to 27 degrees - even 30.

    I was working the ICE - In circuit emulator to the initiated, A vast box of power hungry ECL (emitter coupled logic) and schottky TTL that on a fair day with a following wind could pretend to be an 8086 sufficiently convincingly to be of use debugging the firmware.

    Except when it wasn't. One day code that had previously run well, stopped. Analysis of breakpoint audit trails showed nonsensical behaviour. Opcodes were simply not being executed....as they should.

    I reported to my boss 'The ICE has gone west' .

    "Why?"

    "Dunno. It is getting pretty hot. "

    And we grabbed the manual and leafed to the back

    OPERATING TEMPERATURE.15C-27C AMBIENT

    Now I came from a hardware background, and I knew as well as anyone that those chips had capabilities to go over 100C without popping, and the thing was fan cooled, so the only reason they would have the temperature rating so low was that it was on the bleeding edge built out of selected chips and the timings would not hold out beyond that.

    The thermometer one of the permies had installed on the wall said 28C

    "Leave it with me" said the Boss

    The next Monday we arrived to find air conditioning had been installed.

    Its always nice to know management values its ICE more than it does its coders

    1. Voyna i Mor Silver badge

      Re: Once upon a time

      "Now I came from a hardware background, and I knew as well as anyone that those chips had capabilities to go over 100C without popping"

      Ah, ECL. Runs hot but the problem is, as you note, that being non-saturating its parameters are very temperature sensitive, so unless you have an expensive piece of kit with cooling, the temperature range is limited.

      IIRC there was a Nat Semi ECL minicomputer which had a self test routine where the tail voltage and frequency was adjusted for the CPU periodically.

    2. John Brown (no body) Silver badge

      Re: Once upon a time

      "Its always nice to know management values its ICE more than it does its coders"

      Bloody wimps whinging about heat in a room full of ICE!!

  9. Keith Oborn

    Some years ago-

    At a large machine room in Tokyo owned by a well known consumer electronics manufacturer whose (short) name begins with "S":

    A long row of very heavily loaded racks, with front and back doors off, and in front of each one a standard tall office fan set to maximum.

    The entire division's operations depended on those fans working.

    1. Yet Another Anonymous coward Silver badge

      Re: Some years ago-

      Hopefully all plugged into a series of daisy-chained extension leads back to a single wall socket ?

  10. Andytug

    Never mind the H&S of attaching fans with cables

    If that's a serial cable (as used for attaching laptop to back of switch for maintenance) then at our place it would be a security breach as could be used to access, reset config etc.

    1. steamrunner

      Re: Never mind the H&S of attaching fans with cables

      If someone can get close enough to plug a serial cable into your switches, you have a more serious problem than unauthorised console port access.

    2. Stoneshop Silver badge
      FAIL

      Re: Never mind the H&S of attaching fans with cables

      If someone can access the equipment at all you're just waiting for a security breach.anyway, cable or not.

      1. Korev Silver badge
        Coat

        Re: Never mind the H&S of attaching fans with cables

        But they need to err 9 pin it on someone

  11. wyatt

    Glad they're using fans.

    Customer 1) 6 servers in a converted kitchen with 1 extractor fan (there due to the previous function) taking heat out. Warned on many occasion they wouldn't be supported due to overheating which did end up costing them for repairs to overheated kit.

    Customer 2) Survey completed in a partitioned off area to add a server. No aircon, room was already warm to hot and they wanted to add more heat? Advised they'd need to provide cooling before we'd be able to provide a quote which they wouldn't.

    Customer 3) our server has gone down. On site and no air con, it had failed months ago. Manglement had refused to pay until we refused to fix (at their cost..).

    Seems there's a trend with cooling not being taken as seriously as it should.

    1. Voyna i Mor Silver badge

      "Seems there's a trend with cooling not being taken as seriously as it should."

      All time favourite? Company with managed sir conditioning system. The system developed a fault in the server room one night, the technician turned up to repair the fault...to find nobody there because only the IT department could get into the server room and none of them answered the call.

      It was an expensive mistake.

      1. Brad Ackerman
        Devil

        Re: "Seems there's a trend with cooling not being taken as seriously as it should."

        The fire department can open that door faster anyhow.

      2. StargateSg7 Bronze badge

        Re: "Seems there's a trend with cooling not being taken as seriously as it should."

        The BEST company server room I have ever seen is a local one in Vancouver, Canada which I had a tour of last year. It used a Silicone Oil liquid immersion system. They rack many quad-cpu or eight-way SuperMicro motherboards into what looks like large home refrigerators but are in fact filled with silicone oil which is NON-CONDUCTIVE and is allowed direct contact with the electronics of the motherboards including the GPU's, RAM, DRIVES, etc. A 16 kilowatt power supply (ALSO submerged!) powers each large box which is about 2 metres tall by one meter wide by 75 cm deep. The silicone oil is pumped into the bottom and flows up through each VERTICALLY PLACED motherboard and comes out the top warmer than it was at the bottom. A external radiator and condensor assembly uses huge radiator fins on the roof of the warehouse-like building to get rid of the heat which then recools the silicone oil which is pumped back to the bottom of each sealed rack system for reuse. Since BC Hydro supplies electricity as cheap as 8 cents per kw/hr if you buy blocks of it up front, their mains cost are VERY LOW compared to other parts of the world which is WHY they are located in the Vancouver area!

        There were 150 racks containing 12 motherboards each with over 7200 8-core CPU's (that's 57600 CORES!) doing fast algorithmic day-trading. They could afford it! I was told by a 3rd party that they were pulling in over Two Million Canadian Dollars GROSS RECEIPTS A DAY 24/7/365 for the less than 10 shareholders...NOW EVEN WITH TAXES AND EXPENSES THAT IS STILL A MONSTER INCOME from a bunch of CPU's dipped in silicone cooling oil!

        Some people take IT cooling VERY VERY SERIOUSLY!

    2. tfewster Silver badge
      Facepalm

      I've seen many repurposed broom cupboards. Plus a specialised computer room, which would have been great if the computers hadn't been moved in before it was finished. The servers were lifted (and dropped) by the contractors laying the floor covering. And covered in little piles of brick dust where shelves were being put in.

      One of my employers saw the light, and moved the servers and noisy high speed line-printers out of the general office to their own room. With not just an extractor fan, but external air from the cool side of the building sucked in! Unfortunately wasps built a nest near the intake one year, and we had a computer room full of dead wasps.

    3. Yet Another Anonymous coward Silver badge

      Our customer partitioned off part of the kitchen for the server racks, no AC or extractor fan but lots of steam, fat and smoke - because the printers were on the other side of the wall and so they could just knock holes in it for the cables.

  12. Anonymous Coward
    Anonymous Coward

    FAKE NEWS!

    That "The Bristol Air Scoop" corrugated paper box should have the markings of Amazon on it ...

    It's the little details that give it away...

  13. Giovani Tapini

    I've done this too

    Over 20 years ago, so no smartphone pics.

    Aircon popped in computer room. Getting toasty rapidly.

    Prop open all the fire doors etc, grab, commandeer, steal deskfans from all over the place, step carefully over the spaghetti of cables and watch is roar. It worked, just about, for long enough. Back in the day of the really big hard drives without all these fine tolerances :)

    although there was another occasion (minus the fans) in a different computer room where aircon poppage caused the heat buildup to be so intense that ingress to the room to switch off the old minicomputer was itself tricky and required hands to be covered with jumper to execute. This resulted in melted 8" boot floppy, and I am sure you could have cooked breakfast on the top of the racks. took a day or so to cool down! The drives were built like engine blocks and held their heat nicely!

    1. Stoneshop Silver badge
      Flame

      Re: I've done this too

      Over 20 years ago, so no smartphone pics.

      Aircon popped in computer room. Getting toasty rapidly.

      Close to 35, in my case: I'm in the computer room for some reason when I hear the dull roar soundscape changing. After a couple of seconds it dawns on me that it's the aircon rumble that's absent now. Which means that there's now close to 100kW going in every second that's not taken out, which is clearly suboptimal for the continued operation of the equipment present. I sprint out the door and into the sysadmin pen, and alert those present. Half get assigned to raiding the offices and confiscating any fan they see, the others shut down and switch off all the systems and storage not deemed absolutely essential. It's the one time I've seen a thermograph move, climbing 10 degrees in as many minutes.

      We lost maybe three RA81 HDAs and one logic board out of well over a hundred, and one memory board from one of the systems.

      1. GlenP Silver badge

        Re: I've done this too

        We lost maybe three RA81 HDAs and one logic board out of well over a hundred, and one memory board from one of the systems.

        It was tough kit! Once had a VT-240 terminal that had been in a fire (sadly the VAXes didn't survive). The case had partly melted but when we powered it up it still came up with a reassuring "VT-240 OK"

      2. Alan Brown Silver badge

        Re: I've done this too

        "After a couple of seconds it dawns on me that it's the aircon rumble that's absent now. Which means that there's now close to 100kW going in every second that's not taken out"

        You really _REALLY_ have to wonder why with millions of squids worth of kit and redundancy up the wazoo, manglement cut corners and allow a single point of failure for the cooling.

        1. Stoneshop Silver badge

          Re: I've done this too

          You really _REALLY_ have to wonder why with millions of squids worth of kit and redundancy up the wazoo,

          Eh, what? Redundancy? At that time there were a dozen 785/8600/750 systems, half of them in a cluster, the others standalone (although the two PDPs could run the other's tasks if one of them failed) plus a scattering of MVII's. Disks: lots. Four HSC50s, maxed out, for the cluster. Plus some RAs for the standalones. PDPs ran off 4 RP06es each. I don't think there was disk mirroring in use (can't be arsed to look up if that was even an option with VMS 4.5), but there were a couple of unused disks which were used as lukewarm spares when needed. They didn't even have a no-break until the site became crucial as a comms hub for NW Europe. Which was close to a year later.

    2. Doctor Syntax Silver badge

      Re: I've done this too

      "Back in the day of the really big hard drives without all these fine tolerances"

      I've seen a similar thing but back in the day of massive optical disk library which did have fine tolerances. I think the robot had to be recalibrated.

    3. Chris Evans

      Re: I've done this too

      "Over 20 years ago, so no smartphone pics."

      Back in the mist of time you get a gizmo that was normally black with silver dials and buttons on it about the size of a smartphone but a lot thicker. They were called Film Cameras! I wonder what ever happened to them...

      1. Anonymous Coward
        Anonymous Coward

        Re: Film cameras

        You’ll recall that film cameras have no more than 36 photos available before the SD card, sorry, film canister, needs to be ejected and then developed.

        So, we weren’t really in the habit of snapping every random thing that took our fancy, with the gay abandon that we are today.

        Every frame was sacred!

      2. John Brown (no body) Silver badge

        Re: I've done this too

        "They were called Film Cameras!"

        No, they were just called Cameras. There was no other type until those new fangled electronic Digital Cameras came along :-p

        1. Richard 12 Silver badge

          Re: I've done this too

          A canister full of film with 36 photographs!?

          You youngsters with your new-fangled "film" cameras don't know you're born!

          One glass sheet is all you used to get.

    4. prodromos65

      Re: I've done this too

      Got a late night call out on New Years Eve as the Comms room for a Law firm in one of the buildings we looked after had gone down. It turned out that they had scheduled maintenance on the Comms room cooling system that night as network traffic was at a minimum. In order to keep things cool while the crack units got serviced, several portable A/C units had been brought into the Comms room which the A/C contractor had plugged into power outlets on the wall of Comms room, all of which were backed up by the Comms room UPS.

      Queue UPS overload alarm followed by immediate power loss to all computer equipment in the room.

      It took me a while to get there as the CBD was closed to all traffic on New Years Eve and I had to walk from a couple of kilometers away, so by the time I got there they had already figured out their mistake and had everything running again.

      Had another callout to a bank branch that was suffering from heat related issues. The A/C plant on the roof had broken down so an A/C company had brought in portable units to keep things cool, with the hot air being pumped into the false ceiling which acted as the return air of the currently not working A/C plant. Thus, all the hot air came flowing out of the return air vents in the ceiling and the place just got hotter and hotter, since no one had actually thought about how they were actually going to remove the heat from the building. Their computers were already slowing down due to CPU throttling and it would not have been long before their network equipment shutdown.

      I explained the basic thermal dynamics to the manager, and the fact that they would soon be unable to trade when their network died. They craned a large temporary unit onto the roof the next day.

  14. Anonymous Coward
    Anonymous Coward

    At my old place, management had declined requests for aircon in the equipment room - apparently they considered the existing arrangement, whereby hot air was simply fanned/ducted into the gents' loo next door, was a perfectly adequate solution.

    During a particularly hot summer a piece of melted/distorted hardware was ceremoniously dropped on the MDs desk with the statement "that used to be part of a business-critical server, and that's why we need aircon"

    1. CrazyOldCatMan Silver badge

      management had declined requests for aircon in the equipment room

      One place I was at we were combining two computer rooms into a new building. I'd (privately) done the power and heat calculations and so requested a certain UPS and aircon setup. Unfortunately, they'd already bought the UPS and aircon setup as part of the building fit-out.

      Both of which were nowhere near enough. They would have been OK for either of the old computer rooms but not for the new, merged, room.

      By the time we got to 60% of the move the UPS went into power bypass because the power draw was too high. And because of the room configuration, we couldn't add extra capacity even the the UP was expandable. And the aircon was a two-chiller design that, in an ideal world, would have let the whole room run off just one chiller so that we could take the other one down for maintenance. But again, we had to run both units of full to dissipate the heat - and soon discovered that the 'architect' who had designed the room had put the chiller drip-tray drainage above the server racks..

      Not one of the finest fit-outs that I was involved with - I left that place two months later.

      1. Alan Brown Silver badge

        "Both of which were nowhere near enough. They would have been OK for either of the old computer rooms but not for the new, merged, room."

        BTDT. The result was a server room which was maxxed out thermally at 1/3 it's physical capacity with manglement wanting to put desks in there. Explaining the legal ramifications of putting people in with _lethal_ fire suppression kit made them go a little white. ("manslaughter charges" were mentioned)

  15. Anonymous Coward
    Anonymous Coward

    Desperate times demand desperate measures

    Where I work earlier this year, at the beginning of the heatwave, the machine room just below where I sit suffered a total air conditioning failure.

    By the time someone followed up the alerts, the ambient temperature was heading towards 40oC. What it was inside the racks we don't know, but it would have been much higher.

    As we had just recovered from a major storage failure a few months earlier, and did not want to have to do the same again, the managers put everyone on an emergency footing, but decided not to do an EPO, but wanted a rapid ordered power down.

    Well, that made a very busy afternoon, shutting down 100+ servers and the associated comms. and storage gear in the correct sequence, and in order to try to slow the rise in temperature, every single desk fan was commandeered, as was the portable air-con unit that was keeping the coffee shop cool. All the doors and windows in the corridors around the machine room were wedged open, and security were told to ignore all of the warnings they were getting. The managers themselves manned the doors and open windows to make sure that nobody tried to get in, while we did the technical work.

    Fortunately, most of the systems in the room were for test and development, so the external facing systems were not affected.

    Never seen the managers, project managers and PMAs do so much physical work!

    It must have worked, because we did not suffer any more than the normal number of failed disks following a power down, and in general, it all came back cleanly once the air-con. was fixed. Took a lot longer to bring up than it did to shut down, though.

    1. Anonymous Coward
      Anonymous Coward

      Never seen the managers, project managers and PMAs do work!

      FTFY!

      1. Sir Runcible Spoon Silver badge
        Devil

        Re: Never seen the managers, project managers and PMAs do work!

        Rapid power down of 100+ servers -> Breaker Switch >:)

    2. Alan Brown Silver badge

      Re: Desperate times demand desperate measures

      "By the time someone followed up the alerts, the ambient temperature was heading towards 40oC"

      Our rooms have an emergency power crowbar at 38C for _exactly_ this reason.

  16. Stoneshop Silver badge
    Facepalm

    Additional cooling

    One contract I had was with a branch of the Dutch Ministry of Agriculture, housed in (then) 40-year old barracks. Single story, low tarpaper roof, and even on just moderately sunny days the offices were already uncomfortably toasty. Cooling to the computer room was half-knackered, as one of the two units had a pinhole leak that apparently wasn't fixable, which had caused them to use up their freon-based coolant quota already. This lack of cooling caused serious swings in the computer room temperature, with a resultant well-higher than average component failure rate.

    Their method of trying to keep the temperature within acceptable values was by installing garden sprinklers underneath the heat exchangers, and turning them on when temps in the computer room hit 25 degrees. Time to turn the tap off was often well into the evening, requiring one of the sysadmins to clock several hours of overtime. On really warm days boosting the cooling capacity like this was insufficient, and relief had to be brought by opening the back door and pointing half a dozen floor-standing fans at it.

    (My suggestion to dump a few buckets of white paint on the roof, or tack a couple of rolls of alu-coated foil over it was dismissed, as 'this is just a temporary building'. It had been temporary since the mid-1950's, and it continued to be temporary for at least five more years after I left)

    1. This post has been deleted by its author

    2. H in The Hague Silver badge

      Re: Additional cooling

      "My suggestion to dump a few buckets of white paint on the roof, or tack a couple of rolls of alu-coated foil ..."

      The top floor of H Mansion has a flat roof with black bitumen roofing felt. That floor used to get uncomfortably hot but I assumed that most of the heat came in through the windows and not through the roof as that's insulated. But applied special aluminium bitumen paint to it last week and it's made a real difference. Unfortunately I didn't measure the before and after temperatures.

      Tip: paint the roof before the weather gets too hot. In hot weather the primer, even when dry, is very sticky and that makes work difficult.

  17. Peter2 Silver badge

    . . .

    Ok. Racks get hot because the equipment in them draws cold air from the front, and then blows hot air out of the back. As wallmount cabling cabinets are poorly designed for airflow the hot air in the back can escape in two ways, firstly through escaping out of the tiny holes drilled in the side which probably wouldn't let enough air in to keep a hampster alive, and secondly by escaping out of the front in the gaps around the mounting rails. This hot air then mostly gets sucked back in the air intakes at the front of the equipment in the rack. The rack then gets hotter and hotter until the equipment generating the heat melts.

    We can all agree that's the problem? Cool.

    So why in the name of $DEITY do people blow cold air into the hot aisle they've created to cool the cabinet down? It's just pushing the hot air out the front, hopefully faster than it's being pulled back in again. A more elegant solution is to pull the air out of the hot aisle so that cool air is sucked in the front and hot air goes out of the back.

    A PC case fan is ideal for this, and most IT departments have at least one dead PC waiting to be properly disposed of that can part with it's case fan. Figuring out how to get 12v to it should be an exercise left to the imagination of the reader, but I shall comment that most supermarkets will sell 12v PSU's for a few quid. This combination provably lasts years running 24/7 as long as the H&S people don't see the spliced wiring connecting these two parts.

    There are of course other methods such as using transformers already owned by your organisation, such as running a long cable from the 12v rail on a nearby PC, buggering the dead PC's PSU so that it'll run without anything connected, sacrificing one of those 12v transformers in that box of surplus to requirement parts saved for a rainy day, or wiring 4 (12v) fans from the (48v) output from the PoE switch which is generating most of the heat in the cabinet for Cooling Over Ethernet, but a dedicated 12v supply is by far the best option.

    1. Sir Runcible Spoon Silver badge

      In my experience cold air is usually pumped into the underfloor space, allowing the racks to vent warm air out the top and drawing the cooler air in from below.

      1. Peter2 Silver badge

        With a half dozen top mounted case fans in fullsize cabinets, yeah. Bog standard arrangement.

        But the article was showing pictures of those little wallmount comms cabinets which are generally mounted as close to the ceiling as the cabling contractor could get, which means you can't get access to the top. The article showed people having the same heat management problems with the comms cabinets as with larger cabinets and trying to control the temp by blowing ambient air into the hot space at the back with a normal deskfan.

        My point is simply that controlling tempretures in a small cabinet is best done in exactly the same way as with a larger cabinet; kick the hot air out with a case fan or two and cooler air gets sucked in the other opening.

  18. Chris Miller

    August 1990 broke all sorts of UK records for high temperatures. I was responsible for mainframe systems across two locations, both with well-specified (we thought) aircon units. We started getting over temperature alarms from the backup system, and when we checked we found that temperatures in the sun on the roof (the computer room was in the basement of a multistory office block) were pushing 50C. As a result the heat exchangers were actually heating the water coming from the machine room rather than cooling it.

    Fortunately this was on a Friday afternoon, and though we had a hectic weekend sourcing additional aircon (like hens' teeth, because of the high temps across the country), but the weather had broken by Monday. (We sunsequently replaced the 'dry' heat exchangers on the roof with 'wet' ones that use water evaporation to provide additional cooling.)

    1. The Real Tony Smith

      I've had similar and found that a hosepipe spraying a fine mist of water over the heat exchangers worked well.

      That and removing all the debris from the (horizontally mounted) cooling grids!

  19. spold Bronze badge
    Facepalm

    Just blow on it

    Once reviewed a Health Testing laboratory IT room. While the main servers had backup generators for power failures the Air Conditioning units didn't. Ah no problem they said - we have some big household fans we can bring in.... OK so it's a power failure... where are you intending to plug them in?

    Not the largest problem really... the fridges for the medical testing specimens had no backup power so things would get rather smelly rather quickly. And what was the point of the IT kit running when they had nothing to test...

    1. Ozzard

      Re: Just blow on it

      Yeah, our -80C freezers in the lab are on the UPS and, for bonus points, have big CO2 cylinders behind that they can vent over several hours to keep cool.

      Of course, at that point the entire floor's full of carbon dioxide and we don't have breathing equipment...

  20. SirDigalot

    cooling is for wimps!

    I have an 8 port POE cisco pro-sumer gigabit prosumer switch acting as my core switch in my (unconditioned) loft/attic here is toasty old Florida, solid as a rock, the temps up there get well over 100 F every day, been there untouched for a couple of years now, i know it is still working because if it stops i get text alerts at work from the W.E.W.S (Wife Early Warning System ) and also more alerts from the U.K.W.N.I.S (Upset Kid With No Internet System)

    1. Korev Silver badge

      Re: cooling is for wimps!

      I used to work somewhere with a similar system called Clive. No matter what we did to enhance alerting we still weren't as fast as him.

  21. Kenny_10_Bellys

    Aircon fail

    I was called out last Sunday evening when our refinery monitoring systems went dark. The links between the monitoring network and normal data network turn out to rely on 18 year old Cisco kit that had overheated when the aircon in a server room failed. Cue the open windows and as many stand up fans as I could find. What fun!

  22. saxicola

    The thing with fans is that they just move air about, (and warm it whilst doing so) they don't cool anything. They work on humans because we sweat and thus cool by evaporation of the sweat, moving air helps this process. AFAIK servers don't sweat.

    1. Paul Shirley

      True but hot air tends not to leave enclosed spaces very fast without being nudged and cool air prefers to kip on the floor instead of working :)

  23. Mike Lewis

    You can use picture hooks for cooling

    https://i.imgur.com/vo2zlBm.jpg

    1. Martin-73 Silver badge

      Re: You can use picture hooks for cooling

      @Mike Lewis That's actually the kind of sensible thinking that the designers should have had.

      Another method is the cheap poundland laptop cooler pads that are run via USB... often you can power them from the device itself

  24. razorfishsl

    Clearly some of these "tech" people are complete fucking morons.

    Those boxes have Twin mains powered fans in the top or the ability to mount them.

    And yet they mount the box almost flush to the ceiling.

    We run kit in factories in China same box fronts closed IN FACTORY PRODUCTION LINES

    >40deg ambient

    >95% humidity.

    and yet we don't need to engage in this sort of fuck feast.

    Mind you...... we do ensure that they are checked weekly for dust blockages and that they all have functioning fans.

  25. sanmigueelbeer Silver badge

    The Federal Government agency I worked for before had business units that operate in two ways: Fiefdoms and JFDI (just f*ckin' do it). Fiefdoms came in form of putting physical barriers (like physical walls) between their business units and "others". So a floor of a building will have several walled "gardens" and swipe card access limited to a few. JFDI is followed by "... or else ..." (a career ending gossip). Combine the first and the second and you'll get a business unit doing something without the other business units knowing about it (particularly IT). Duplicate work? Yeah, tons of it. The role of IT was to "filter" those duplicate work and tell the other work that "`tis been done already by <enter business unit here>." Also, I'd like to emphasize that with JFDI, business units don't like "change management" or "change control".

    So back in 2001, a business decided to wall off a quiet corner of an underground car park (two floors below the street). The business unit decided that they wanted to put some servers. They put a tiny a/c unit in this space and it worked fine. Over the next 5 years, other business units caught wind of this scheme and, without consulting anyone, started putting their own servers in. This space grew from one rack to >12 racks (each side) and then one day, someone christened it a "data centre".

    So it is of no surprise that during winter time, when the car park temperature was around 5C the inside of the "data centre" was hovering at >25C (both chillers running at 100% > 4 years without proper maintenance).

    One day, one of the chillers decided to call it quits. Within 30 minutes, it was a full blown mess (servers auto shutdown and all boards lit up with a sea of red). It was >35C at the entrance (and hotter in the middle). The a/c tech said there is no way to source the parts in Australia and Asia. It took MONTHS for the CIO (yup, he had to sign off on that due to the cost involved) to authorize new chillers.

    To keep the place down the two doors were left open, a full-time security guard at both entrance, during winter and blower fans were used during the summer.

    Fun times, that was.

  26. Mandoscottie

    2nd pic

    Utter genius........that has to be the best use of an old RS232 Cisco management cable ever :D

    1. keith_w

      Re: 2nd pic

      "Utter genius........that has to be the best use of an old RS232 Cisco management cable ever :D"

      What old? Looked like the one I used fresh out of the box a couple of weeks ago.

  27. Paul Shirley

    shoelace, when cable ties won't work

    Instant flashback to the many jerry built cpu cooling solutions I've used down the years. The 12cm cooling can tied onto a fan 12to8 reducer and stupidly heavy copper radial heatsink with a shoelace worked surprisingly well. As in didn't fall off or break the vertically mounted mboard or CPU holder.

    Still always a surprise how well popping the case and aiming a deskfan at the mboard works as well.

  28. Anonymous Coward
    Anonymous Coward

    Pffft Amateurs

    The professional way is to over spec the cooling system and then use it as a fridge overflow and/or wine cellar.

    1. Ozzard

      Re: Pffft Amateurs

      My father worked as a maintenance engineer on "The World's First Supercomputer" - Atlas (see https://www.youtube.com/watch?v=6TRfy70DqD8 for some of his comments on modern washing machines). The machine cooling made a really good fridge for the (then new) Ski yoghurts.

      1. prodromos65

        Re: Pffft Amateurs

        A lecturer I had at university commented that the old reel to reel tape drives in use in the computer rooms each had their own cooling system, and there was a space in the back of each which was just big enough to fit a six pack of beer.

  29. Rohime

    But isn't the outside temperature like ...

    26 C or something? That's hardly even warm (says the Australian)

    1. Alan Brown Silver badge

      Re: But isn't the outside temperature like ...

      "That's hardly even warm (says the Australian)"

      The Australian isn't getting 18 hours of daylight coming from 270 (horizontal) degrees of sky. It gets a little tricky to shield against the direct heating effects of this kind of thing. (suprisingly, it's _much_ easier to work against near-direct overhead sunlight at low-latitudes than low-incident angles at high latitudes. on the other hand you only need to deal with this shit for 2-3 months of the year)

  30. Anonymous Coward
    Anonymous Coward

    Suck air out of rack and blow into another room

    I always add an inline good quality 150mm fan plus ducting to suck air out of rack and blow it into a nearby corridor. It's the circulation of air that helps, even if the air coming in is already warm.

    For if fan fails I also get an Electrician to add a temp alarm that can be muted but with a red light, with instructions for staff.

    Done this at many sites for many years, cheap and works really well. For those of you with selfish ceos, pipe it into their office and do your farting by the rack.

  31. xehpuk

    Soda cans

    I heard some guys cooled their chips with soda cans. It may have been something with probes being in the way for the cooler.

  32. My Alter Ego

    Not as Heath Robinson as some

    We ordered some ali box sections to make a 1.2x1.2x2.4m box that fits around our server cabinet. Picked some 25mm insulation board from Wickes, made some brackets (rivnuts are awesome) to keep them in, cut a slot and shoved a window AC unit on a shelf dumping in cool air.

    Keeps the interior of the cabinet at a nice 20-23°C, while the outside can get up to 35. AC runs through our UPS so if there is a power cut it'll run as long as the cabinet does.

    I kept the fact we have an AC unit quiet, in case the meat sacks get jealous!

  33. Thomas Martin

    I would fire some employees

    If our network wiring looked like that, I would fire the network manager. Heaven help you if you had to trace a cable. Whoever did that had no idea about network cabling.

  34. 10forcash Bronze badge

    Back in the day (2002) we had an IT tent, complete with a pair of DL380's and associated kit. ambient was 38 Celsius at it's peak and the wind direction determined the server maintenance schedule (daily or weekly). Still, could've been worse... and lo, two years later, it was. 55 Celsius, no time for acclimatisation and queues for the portaloos (emptied at 1500 daily). - but at least they were civilised enough to have British mains sockets!

  35. Anonymous Coward
    Anonymous Coward

    1970s vintage Telecoms transmission and multiplexing equipment at my place of work is kept cool with the aid of the pedestal fan that used to keep me cool at my desk! It’s a vital piece of the nations infrastructure and is only kept alive by a £20 fan from Argos.

    The UK; investing for the future!

  36. mr_souter_Working

    Too Hot = bad - too cold = bad

    previous company I worked for (small IT company, did some maintenance and support) - one of the customers had a small dell server sitting under the stairs in their very old building.

    got a call one morning - went out to it, to discover that it had shut down overnight due to the cold (pretty cold winter in Scotland).

    a big blanket to trap the heat in it's little nook, and a heater nearby - it was fine - they were advised that it should be in a temperature controlled area, but the scottish newsreaders that owned the company probably never did anything about it.

    at the same time. the company that I worked for moved into new offices in Glasgow - with no aircon in the server room - the solution proposed by my boss (he wanted to reuse the waste heat from the servers to warm the office) - was to put a small desktop aircon unit on the shelf in front of the only window in the room, and open the window - servers regularly hit 50C during summer months - and shut down more than once while I was there.

    A previous company to that, one of our customers had their servers in a hall cupboard - just about every night the servers would shutoff due to heat, and come back on in the morning - they were always complaining about the systems being down in the morning - turns out they were closing the cupboard door when they left, and opening it when they came in - no airflow, heat built up - servers shut down.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019