back to article Your servers are underwater? Chill out – liquid's cool

Every time we tweet, procrastinate by watching an online video of a puppy with hiccups, or query a cloud, we spin up a chain reaction of hardware and electrons in some datacenter somewhere. This generates heat that must be dissipated. Moore’s Law – the observation that recently celebrated a 50-year milestone and which …

  1. Voland's right hand Silver badge

    Minor problem

    Capacitors :(

    Most capacitor designs are intended for operation in air at ambient pressure. Dipping capacitors into liquid coolant really screws that up.

    So you either need a very special design where you pull the voltage regulators and power supplies outside the main coolant dip or you have to throw out your mainboard and power supply every 6 months.

    1. TheVogon

      Re: Minor problem

      "Most capacitor designs are intended for operation in air at ambient pressure. Dipping capacitors into liquid coolant really screws that up."

      It takes about 10 metres depth of water density coolant to add 1 atmosphere of pressure - so unless you are thinking of James Bond movies style nuclear reactor cooling tanks, that's really not an issue at all...

      However the highest efficiency future datacentre designs generate power directly in the rack - e.g. from gas catalyst fuel cells, etc. - and these systems are already at the correct voltage - so minimal power supply electronics are required...

      http://www.datacenterdynamics.com/critical-environment/microsoft-our-in-rack-data-center-fuel-cell-concept-works/84945.fullarticle

      1. MacroRodent

        hard drives?

        What about hard drives? I think they still have a hole with a filter for equalizing air pressure inside the shell, and will certainly not work if full of oil. Of course not an issue if all your storage is solid state.

        1. cray74

          Re: hard drives?

          Puget Systems, which makes mineral-cooled kits for home PCs, has tested the idea of submerging hard drives in mineral oil. As with fish, mineral oil submersion is poor for hard drives:

          https://www.pugetsystems.com/blog/2013/09/09/Can-you-submerge-a-hard-drive-in-mineral-oil-501/

          1. razorfishsl

            Re: hard drives?

            Since most drive heads require a layer of air to float, adding a viscus liquid is really going to mess things up.

            not to mention the torque on the motor of grinding a disk surface against a liquid

        2. Brandon 2

          Re: hard drives?

          I thought the guts were sealed and filled with nitrogen...

          1. AndrueC Silver badge
            Boffin

            Re: hard drives?

            I thought the guts were sealed and filled with nitrogen...

            Nope. They've always been open to the air with filters to trap particles. Back when I was in data recovery the engineers would have drives spinning with their tops off for days in their clean room. We even had one on a stand at a trade show for a few days. The trouble with sealing them is dealing with the changing pressures especially comparing pressures in an aircraft hold with operating pressures nearer sea level.

          2. Gene Cash Silver badge

            Re: hard drives?

            You're probably thinking of the new drives that are filled with helium:

            http://www.theregister.co.uk/2015/03/25/lighterthanair_drives_floating_to_top_of_hgst_heap/

            I'll bet those have no problem running in oil, and I wonder if you'll be able to see the bubbles of helium leaks?

        3. Suricou Raven

          Re: hard drives?

          The He6 has a niche then.

        4. Gazareth

          Re: hard drives?

          Don't think the drives need cooling particularly. Or at least, not to the level that would require liquid cooling.

          1. razorfishsl

            Re: hard drives?

            yes they do.....

            Take a raid 5 array, run a rebuild.... and see just how much cooling they need.

    2. Olius

      Re: Minor problem

      hahaha, I don't think we would be reading an article about this tech if it didn't already work ;-)

      1. Charles Manning

        Please consider the upside

        If you're a BOFH, or training to be one, you can write off your SCUBA training and equipment as a business expense.

    3. Flocke Kroes Silver badge

      Have you tested this theory?

      Oversimplified:

      The capacitance is εA/d where ε depends on the material, A is the area and d is the distance between electrodes. Increasing the pressure will decrease d, decrease A twice and might do something to ε. Unless you have proper figures from the manufacturer, a small decrease in capacitance with pressure is a sensible guess.

      The maximum voltage depends on d and the material. Increasing the pressure reduces d, but also (often) increases the breakdown voltage. Unless you have proper figures from the manufacturer, expecting the maximum voltage to remain constant is a sensible guess.

      The other figures of merit are impedance at working frequency, maximum current, and temperature for 1000 hours of life. (1000 hours is far too short, so over specify the voltage by a factor of two to double the life, and over specify the temperature by 10C to double the life. Repeat until life time = mandatory guaranty.) It is really hard to guess what some extra pressure would do to these values, but you can be confident that the voltage and temperature markings on the capacitor wildly exceed those it will actually experience, and the whole idea of liquid cooling is to be more effective than air cooling.

      How much pressure: every 10m of water get you an extra atmosphere of pressure. I doubt that the tanks are a whole 2m deep, and oil floats on water, so a sensible guess is less than a 20% increase in pressure.

      In digital circuits, the capacitors are there to stop ground bounce. The capacitors were not selected for capacitance, and ±20% is often chosen because they are cheap. The figure that matters is impedance at some frequency. Often two sizes are fitted to cover a wide range of frequencies. The actual impedance has to be 'low enough', and as the components are cheap, adding plenty is often cheaper than experimenting to find out how few you can get away with.

      In power circuits, the important number is maximum current. Selecting for this normally limits you to capacitors that have more capacitance than your circuit requires. The result is usually harmless, especially as modern (or ten year old) switch mode controllers have soft start built in to deal with large capacitive loads.

      The only thing where pressure will make a clear difference is _after_ the circuit has failed. Abusing electrolytic capacitors causes them to create gasses inside the can until the can ruptures. The cans have been scored so they burst before the internal pressure becomes excessive. Increasing the external pressure will delay the rupture, and a liquid environment will do a better job of transmitting a shock wave to the rest of the power supply. This is entirely academic because the capacitor only ruptured because the power supply was already badly broken.

      I can understand manufacturers voiding warranties as a precaution because they have not tested there components at 1.2 atmospheres. When I have looked for data, manufacturers did not know and did not care. The theory does not point to a clear and obvious problem, so the only way anyone is going to know for sure is to dunk a hundred power supplies, run them for two years and count the survivors.

      1. Gideon 1

        Re: Have you tested this theory?

        Capacitors have more of a problem with low pressures, especially types that do not have a hard case, e.g. resin dipped tantalum. The airworthiness directives are a good guide.

      2. Anonymous Coward
        Anonymous Coward

        Re: Have you tested this theory?

        "The cans have been scored so they burst before the internal pressure becomes excessive."

        So that's the reason for the two cuts across the top of a small SM capacitor can?

        A small buck voltage up-converter module failed recently on first use. The circuit was defined as producing up to 35 volts output from a lower input voltage. The output voltage was selected by a potentiometer.

        The output capacitor was only rated at 35 volts. On initial power-up - after a few seconds the SM capacitor exploded. The can didn't burst though. It was propelled into the air intact - like a bullet.

        It seems likely that the extreme potentiometer setting could produce more than 35 volts. Apparently identical modules from other sellers have a 50 volt rated output capacitor - with a 35 volt one on the input side..

      3. TheVogon

        Re: Have you tested this theory?

        "Increasing the pressure will decrease d"

        No - no it wont. Capacitors for power supplies are generally solid or liquid filled, so not compressible except at extreme pressures. Merely immersing them in low levels of liquid is going to make no measurable difference

        "have not tested there components"

        their

  2. Potemkine Silver badge
  3. Anonymous Coward
    Anonymous Coward

    Making the servers waterproof in the first place would be better as water is more readily available.

    We can then have datacentres in the Thames - The heat used to draw up water through the Underground to cool it and the resulting heat sent to buildings - the pipes are probably still there from the hydropower thing.

    And better for connectivity than an icelandic glacier!

  4. Kubla Cant

    And at the other end of the scale

    I recall reading an article about five years ago, on an overclocking site, where somebody immersed the electronics of his PC in a tank of Mazola or something. Workable but messy.

    1. Dan Wilkie

      Re: And at the other end of the scale

      Wasn't that on Toms Hardware when they were going for their Super-Mega-Ultraclock on an i5 if I recall? Think they were aiming for 5Ghz/Core?

      1. Gordon861

        Re: And at the other end of the scale

        Yes TomsHardware went through a crazy phase of trying to see how far they could overclock stuff.

  5. Joe Drunk

    It must get awfully messy for the poor sod who has to replace a blade/hard drive/memory/power supply in the event of failure.

    1. Anonymous Coward
      Anonymous Coward

      Well I assume at the point it becomes really practical most things will be blades and there'll be an autoloader. Who cares if a drive/memory/power supply/cpu has broke, unload it load in a replacement send the server to a dryer machine then packaging machine, then into the googlecar to drive the day/week/month bad batch to the repair depot.

      Just imagine how quiet one of these data centres would be compared to the data centre of today.

      1. TRT Silver badge

        No swearing as techs slice their fingers open yet again on razor sharp aluminium extrusions because beancounters found a vendor 20% cheaper even if they didn't roll the edges of their stamped out chassis?

        1. Yet Another Anonymous coward Silver badge

          And you would prefer sticking your hands into the chip fryer they are proposing here ?

          1. TRT Silver badge

            I was referring to the techless server centre.

      2. Suricou Raven

        No drying. That sort of coolant doesn't evaporate. You have to let it drip off, and you'll never get the thin slime off completely.

        1. Adam Foxton

          I use oil-cooled electronics daily- it's been common for cooling and waterproofing subsea kit for decades, I'm amazed it's considered such a new idea by you guys- and they clean up nicely. Just remember a non-conductive contact cleaner. Even Limonene or alcohol based ones are good if the equipment won't be powered on for a while!

  6. Anonymous Coward
    Anonymous Coward

    I would have thought the lack of take up was more obvious

    Probably the weight of higher density servers + liquid coolant is beyond a lot of locations floor or indeed building's capability.

    If the cooling liquid is non-conductive and non-corrosive I would not have expected the boards to have a reduced service life in datacenter terms. I trust however that these liquids are non-toxic however...

  7. Cuddles

    Heat capacity

    "a dielectric mineral oil blend called ElectroSafe, an electrical insulator it claims to have 1,200 times more heat capacity by volume than air"

    Not really an amazing claim. Petrol has volumetric heat capacity 1,200 times higher than air, so all they're saying is that their mineral oil is, in fact, oil (specifically short-chain alkanes very slightly longer than the mainly octane and heptane in regular petrol). This also highlights why water is still the preferred substance for actually shipping the heat away with all the faff that having an extra heat exchanger requires - its heat capacity is another 4 times higher again.

    1. Loyal Commenter Silver badge

      Re: Heat capacity

      I seriously hope it's not alkane-based, or you're going to be able to toast marshmallows from half a mile away. It's more likely to be silicone based.

      1. Cuddles

        Re: Heat capacity

        "I seriously hope it's not alkane-based, or you're going to be able to toast marshmallows from half a mile away. It's more likely to be silicone based."

        Nope, mineral oil is (mainly) alkanes. Quite commonly used not just as coolant, but also as heat transfer fluid in consumer electric radiators and in plenty of other places (see Wiki - http://en.wikipedia.org/wiki/Mineral_oil). You probably wouldn't want to use it as a open bath since the fumes could cause issues, but it's pretty safe since it's heavy enough not to be easily flammable - it's around the same weight as diesel, and you can happily drop lit matches in that all day long without setting anything on fire.

    2. Anonymous Coward
      Anonymous Coward

      Do they want a fire?

      Because that's how you get a fire.

      1. DropBear

        Re: Do they want a fire?

        They claim it's non-flammable. Actually, they claim it's safe enough to drink. Not that I'm inclined to test that...

    3. SImon Hobson Bronze badge

      Re: Heat capacity

      > This also highlights why water is still the preferred substance for actually shipping the heat away ...

      Indeed, water is a very good medium for thermal transfer - and if kept pure enough is non-conductive too. In fact, some large generators (think 10s of thousands of volts !) have water cooled coils by the technique of making the copper conductors in the coils from hollow bar and pumping water through them.

      Yes, it does have to be very pure !

  8. Marvin O'Gravel Balloon Face

    The future is Iceland.

    OK, not so great for trading data due to latency, but cheap geothermal power, an endless supply of cold air, and the heat generated can be recycled into domestic/industrial heating and/or energy. Plus the tax breaks are pretty good.

    1. Yet Another Anonymous coward Silver badge

      Re: The future is Iceland.

      If you are going to put all the computers in what is effectively a deep fat fryer - then the future is probably Scotland.

  9. CAPS LOCK

    It's the seventies again ...

    .... hurrah.

  10. Little Mouse

    "The warm output water can then be used in other parts of the business..."

    If the coolant is oil-based you could dissipate the heat by cooking chips.

    1. Andy The Hat Silver badge

      Re: "The warm output water can then be used in other parts of the business..."

      Warm oil baths are good for the skin, apparently, so why not have a beauty salon on site and charge fat, rich people extortionate fees to luxuriate in blade induced warmth - "an ultra high frequency digital toning treatment using futuristic heat pumposoidal techniques and plasmondo-rich liquid drenching"? Warming the treatment rooms, plus washing and drying laundry using the heat exchangers all saves cash ... got a feeling I can make more money out of de-ugly therapy than the data centre itself!

      How much does it cost to strain fat people out of plasmondo oil?

      BTW 'plasmondo' is my trademark which I invented just now for a super beauty product that does magic stuff if you soak bodies in it and, by sheer coincidence, is an inert liquid cooling product sold under another name at one third the price by 3M ...

  11. druck Silver badge
    Mushroom

    Liquid metal cooling

    Liquid sodium as used in fast breeder reactors, burns in air, explodes in water - nice.

    1. DNTP

      Re: Liquid metal cooling

      Just make sure there's plenty of high-def security cameras watching that.

      Not to prevent sodium theft, but so when something goes horribly wrong we can see it on youtube.

    2. Solmyr ibn Wali Barad

      Re: Liquid metal cooling

      Now that's really cool. Or hot.

      /coat.jpg/

  12. Anonymous Coward
    Anonymous Coward

    I'm sorry Dave, I can't go into orbit with that

    I know we usually use ancient computer technology in space, but if liquid cooling became the norm, the gap with earthbound computers might well widen some more.

    Just imagine all the problems with the oil floating around your spaceship if you tried that for the main computer on your long interplanetary voyage and had to deactivate the computer...

    1. cray74

      Re: I'm sorry Dave, I can't go into orbit with that

      If your space hardware needed liquid cooling then you could probably use point-cooling with liquid-cooled heat sinks and dry disconnects.

      I believe the science and electronic racks on the ISS already offers water cooling. While the external radiators use ammonia, heat is collected inside the ISS with water-based coolant through low temperature and medium temperature loops. The medium-temperature loop cools avionics.

      http://www.nasa.gov/pdf/473486main_iss_atcs_overview.pdf

      Going a bit off-topic, the figure on page 14 of the above .pdf illustrates the challenges of cooling in space, which I find interesting. The deployed radiator array with human to scale is a 14-kilowatt radiator. Compare that to the 50- to 300kW radiator of your typical automobile.

      1. ScottAS2
        Headmaster

        Re: I'm sorry Dave, I can't go into orbit with that

        Of course, this is because the ISS radiator is actually a radiator; i.e. it mainly dissipates heat by radiating it. The "radiator" in your car - indeed, most things us Earthbound humans call radiators - are actually more properly convectors, since we have this oh-so handy "air" stuff that you can just dump heat into and allow it to float off. Just goes to show how hard space is.

        Trivia of the day: the Discovery One in 2001: A Space Oddessy was originally going to have enormous cooling radiators for the nuclear reactor at the back, but they were dropped because Kubrick didn't want to have to explain why the (otherwise fairly realistic) non-atmospheric spacecraft design had "wings".

  13. John H Woods Silver badge

    Air cooling has some other problems, too ...

    ... zinc whiskers? Or would that also be a problem with these liquid coolants?

    1. cray74

      Re: Air cooling has some other problems, too ...

      Tin whiskers are a problem in vacuum and air, but may be suppressed with non-ecofriendly options like adding lead to your solder or more environmentally conscious conformal coatings like parylene. Or both.

  14. Anonymous Coward
    Anonymous Coward

    Serious question, how high could I clock a Raspberry Pi in this juice? Considering it normally runs at room temperature without a heatsink or fan.

  15. KaneSama

    BOFH

    This just make be remember this:

    http://www.theregister.co.uk/2002/05/11/bofh_and_the_vax_cluster/

    1. Nunyabiznes

      Re: BOFH

      You beat me to it, congrats. I got sidetracked reading it again...

    2. Yet Another Anonymous coward Silver badge

      Re: BOFH

      Concorde and the ST71 used fuel to cool their airframes - so a diesel cooled cluster isn't a bad idea.....

      1. TeeCee Gold badge

        Re: BOFH

        Do you mean SR71?

        That had the interesting feature that after takeoff and climb to altitude it then went through in-flight refuelling.

        The fuel was kept in the wings without the benefit of tanks to save weight. Until the skin warmed up due to air friction and expanded, sealing the joints, it leaked like a sieve. Thus the drill was to get it up to the maximum altitude of a laden tanker aircraft, losing most of its fuel on the way due to leakage, refill and then immediately wind the thing up to full chat to get its outside nice 'n hot and keep the second lot of fuel where it belonged.

        Presumably smoking in the hangers of Area 51 was frowned upon......

        1. Vic

          Re: BOFH

          Presumably smoking in the hangers of Area 51 was frowned upon......

          Apparently not.

          According to a documentary I watched the other night, the SR71 had a special fuel which would not ignite if you threw a lit cigarette into it.

          This kinda makes sense, what with the amount of juice drippiung off the wings as it took off...

          Vic.

  16. Wommit
    FAIL

    Well, apart from this article sounding quite a lot like an advert, I can see a number of problems with this.

    Weight, as has been mentioned above, most current buildings won't be able to take the weight of this type of oil bath. Ground floor, mmm... maybe, with a good thick concrete pad. . First / higher floors, no way. Greenfield sites would be able to build the necessary infrastructure, but the cost might put a been counter off. Also the infrastructure required to manage the oil and it's subsequent water / air cooling will be complex, Involving skills not yet normal in a data centre. The weight of an oil filled server is going to be a problem too. (Yes, I know that servers can drain, but you have to raise it out of the oil bath to allow that.)

    Contamination, muck and debris from the blades & components themselves would be an unwelcome intruder into an oil baths circulation system. Replenishing / replacing oil is not usually a clean job.

    Repairs & replacements are going to be a nightmare. Taking any server or Blade rack out of the oil bath is going to very messy. It is also going to take much more room to operate on the oil bath(s) than that of a current air cooled rack space. Another cause for oil replacement, And what about the units warranty when returning a (new-ish) component for replacement? Upgrading, well just forget about that. Replace the whole bath with a new unit.

    Fire risks. Now have a quick word with your insurers as to how you're going to minimise the fire risk and how much they'll want to even consider insuring you. Oil burns, electronic components get hot, sometimes very hot. Should the temperature of your oil bath exceed a safe limit the whole bath has to be taken down. How many servers did you put in there?

    Leaks, they're going to be fun. They will happen, usually when most inconvenient. Just what are the health & safety implications of a large number of litres of oil washing round the floor going to be?

    Currently oil baths are fine for one offs, or trial installations, but the technology needs to mature a lot more before it becomes viable for large data centres.

    1. Anonymous Coward
      Anonymous Coward

      https://www.youtube.com/watch?v=Z9_jIa2WADc

    2. Blane Bramble

      All the liquid cooled data centre articles I have seen have missed one very important point. Data racks stack servers vertically to achieve high density for a given floor footprint. Putting servers in a liquid bath would appear to limit you to fewer servers in the same footprint, and that's without taking the extra liquid weight into consideration.

      So, you get a more efficient cooling footprint, but a less efficient density in the first place. Given that power and cooling costs are effectively passed on, what most data centres worry about is density. This increases their energy efficiency (nice tick box, but nothing for the business profitability) and reduces their maximum loading (and therefore potential income).

      1. John Riddoch

        Well, there is the fact that you don't need to worry about hot/cold aisle containment, nor the requisite clearances for doors in front of the racks, so the actual loss of capacity probably isn't as big as you'd first assume.

        I suspect the benefits are more where land is cheap but energy (electricity) is expensive.

  17. Tromos
    Joke

    This goes against conventional wisdom

    Oil is for frying chips not cooling them.

  18. GH1618

    Liquid immersion cooling is old news.

    Seymore Cray put the Cray-2 circuits in an inert cooling bath in the 1980s.

    1. John Savard

      Re: Liquid immersion cooling is old news.

      True, and Novec is a fluorinated liquid - just not a CFC, but instead, I presume, related chemically to the fluids which replaced Freon in refrigerators.

  19. Anonymous Coward
    Anonymous Coward

    Works better now that SSDs have taken over for internal storage

    You wouldn't be able to run your Vmax immersed since the hard drives would allow liquid inside, but that would seriously reduce your cooling footprint.

    I assume you have to get server blades designed to run without fans without complaining, since I'm sure those puny CPU fans wouldn't do well trying to turn against the resistance of a liquid.

  20. wolf2600

    I just hose down the racks every afternoon during the summer.

  21. Boudicca

    bonkers

    Imagine EMC/HP sending their engineer to site, with some Flippers and a snorkel......Jacques Cousteau they are not. EMC/HP Dispatch..calling.....have your emptied the fish tank first before we book the engineer to replace those fault HBA's please, ensure you supply a rubber ring and a life guard just in case.

  22. pardonme

    Thank you!

    I have learned so much from this article and the fascinating comments

  23. a_mu

    capacitors et all

    Well

    I've designed boards that are liquid cooled,

    all the boards are in a big box which is filled with a pink silicon type liquid, and the box closed.

    Yes there is room for expansion, and the walls of the box have water flowing through them to conduct the heat away.

    Biggest problems,

    any oscillators / crystals need special attention

    the impedance of the tracks on the surface of the board are different to being in air.

    ( but as there is no dust / skin cell build up over time, at least its possible to calculate )

    Capacitors not a problem, if you use the right ones.

    The most common capacitor is a ceramic, and most of them are fine,

    just check.

    When it comes to analog / rf, then things are a little more difficult, the fluid does tend to kill them,

    but digital is just fine.

    But

    it does make the thing REAL heavy

    and once a board has been in the fluid for long, its impossible to solder to it ever again,

    the fluid just can't be cleaned off sufficiently well to make a good joint,

    For 'fun' one time, we ran one of the boards in liquid nitrogen,

    now that was cool, but we dare not seal the box !!!

    Will it catch on at home, I don't think so, but some of these real small PC's that fit in the hand might be interesting in a sealed box full of liquid.

  24. Deryk Barker

    "Although liquid cooling is now commonplace in HPC applications, part of the reason for its slow adoption in enterprises and mainstream data centres is often put down to hydrophobia among IT managers."

    I've had my problems with managers, but have yet to meet one with rabies...

    1. Anonymous Coward
      Coat

      rabies?

      managers with rabies, isn't that just normal for work-life?

      having the kit inside a big pool sounds great I'll get my trunks

  25. Dr Dan Holdsworth

    Heat pipes may offer a better solution

    If you connect all the components that generate heat to a simple aluminium heat exchanger at the back of the server unit, and run water through that block, then you get almost the same level of cooling without all the messiness of soaking everything in oils, or the vulnerability and fiddliness of running liquid cooling pipes to each component.

    You would need a second loop to dissipate the heat as the water mix circulating in the pipes would need to be purified water with antibacterial additives, but this isn't really a problem. The second loop could be a simple cooling pond external to the datacentre, or even set up as a moat around the data centre complex. Such a set-up gives the site a nice rustic sort of look, whilst limiting the options of local scallies looking to raid the place.

    1. Paul Hovnanian Silver badge

      Re: Heat pipes may offer a better solution

      This is the best compromise. There are already sime high end gaming machines that I've seen with self contained liquid cooling. It just circulates from the main processor and graphics card to a radiator at the top or back of the case. For racked servers, it would be a simple matter to have an ordering option for a liquid to liquid exchanger. The board components (other than the processor) can be cooled with a small internal fan and heat exchanger to the cooling loop. The air inside the case would be a closed loop, so no dirt and dust.

      Racked servers could be connected to a liquid cooling loop with quick disconnects (similar to what is available for garden hoses). So maintenance, including swapping out an entire blade would not involve going elbow deep into a fish tank. Less coolant would be needed, as most of the server/rack volume would be air. And the coolant would be contained within a loop. So less concern with flammable liquids or weight when adapting to existing building structures.

      1. efestnetz

        Re: Heat pipes may offer a better solution

        Modern RAM and ethernet controllers are not powered by unicorns. They use electrons instead. It definitely is not just the CPU generating heat.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like