IBM smacks rivals with 5.0GHz Power6 beast

Yeah, but...

How does this compare to the new Macs from Apple? Shoddy reporting as usual from Mr. Vance, he never looks at how these so called 'enterprise' systems stack up to the best computers on the planet....what is he afraid of? Is he thinking his overlords at Sun, IBM, and HP will crush him like a bug? Or that they will withhold his monthly stipend?

Is it a power beast as well?

So IBM has to resort to extreme water cooling in order to reduce this thing's power requirements on the data centers they are deployed in.

A) Does this help make for a more "beastly" price compared to a Sun based soution?

B) Even if this cooling solution reduces the power draw by 40% percent over previous Power solutions, is it still more power hungry than a Sun solution?

I'm not a Sun fanboy, just curious. Also very suspicious about what IBM did not reveal as opposed to what they did reveal.

I was in love

Before y'all mentionned Vista.

I might still get one of these shiny racks. That's on my todo list. Just before "buy a Vista-certified box for personal use" (they come at about the same price anyway).

Water cooling P6 adds extra costs and complexity.

Water cooling for CPU's is an admission of a design failure. The extra pipework and electrical power to run the water cooling just adds costs and complexity. Extra electicity is required to run all the water cooling, so there are not savings, this needs to be taken into account and cannot be a hidden infrastructure cost. Plus to add an extra water infrastructure to a computer center with all that electrical wiring is dangerous. Computers and water do not mix well together. Using water cooling for computers is a technological backward step.

All described here: http://blogs.sun.com/ValdisFilks/entry/water_and_electricity_do_not

Water cooled.....

You could also use it as a giant coffee machine.

"Fire up the database Geoff, I need a double skinny latte" (whatever that is).

Water Cooling

Good heavens - water cooling in data centres is dangerous? Well we managed back in the 70s and 80s without too many people being electrocuted on mainframes back in the days of ECL chips. In fact it can make a great deal of sense to use liquid cooling as it is a far more efficient way of carrying high concentrations of heat than is using air (a tad quieter too - the air cooling systems oin some disk arrays can be so loud as to require the use of ear defenders). Note that virtually all car engines are liquid cooled, not air-cooled. Gas cooled nuclear reactors turned out to be a dead end and liquid cooling has won out.

Using water cooling has one considerable advantage - it's possible to export the heat directly outside the building and not have to do through a process of cooling air, pumping the air through the building and then heating it up.

Of course it's better not to generate the heat in the first place, but for some sorts of workloads which cannot be readily split or which depend on single-thread processing speed (read huge, high-thoughput database servers as the prime example) then there often isn't a lot of choice.

I suspect that most IT people don't come across the sort of monster workloads that suit these beasts. There isn't a need for a vast number of them, but for certain applications they most certainly are needed.

Water Cooling

I disagree with Valdis.

Air cooling is inefficient and results in significant power wastage. If the heat can be removed with water to some central point where it can be lost, there will be significant energy savings relative to running CRAC units to keep an arctic datacentre. Water cooling also enables the datacentre power density to be much higher, instead of the rule-of-thumb 4KW per rack which is as much as conventional raised floor cooling can handle without hot spots.

The inhibitor on water cooling is that nobody wants to break ranks first. Kudos to IBM for going first on this. While I remain to be convinced whether cranking Power 6 up to 5GHz will offer any real customer benefits, it's entertaining to watch.

Greener than given credit for?

Surely with so much hot water being circulated, it could be "lost" to the building during winter, thus saving money on the heating bill...

Re: Greener than given credit for?

This could equally be done with an aircooled datacentre. Cambridge University's computer lab can have its waste air pushed through the building's heating system in winter, for example.

Water cooling

A couple points, raised in these comments, which I will try to answer.

1) Yes we used water last century in datacenters. But that is because we did not have a choice. When aircooled systems became available we had a choice and kicked the water cooled computers out very quickly, to be replaced with air cooled computers.

2) Water cooling on occassion failed, leaked, had to be switched off, water cooled servers needed to be switched off. Things got really ugly, I worked in v.large datacenters I saw these problems. Someone mentions that we used water without too many people being electocuted, to me one such water cooling disaster is one too many. Avoidance is a better policy.

3) Air cooled servers could coutinue to run, while water cooled servers needed to be switched off, when the safety guys decided the water leaks were too much of a risk in a high voltage datacenter then we had to switch off everything.

4) Yes water may be a better coolant than air, it adds complexity though. Magnox is a better coolant (shorter half life) in Nuclear reactors than water in PWR. But magnox much more complex to maintain than water. In the same way that water is more complex to manage in a datacenter. Using oil is just another level of complexity.

5) Are we proposing that we justify to buy a computer to help heat the building. When we decommission a water cooled server, do we need to buy extra boilers to heat the building. This is a plumbing nightmare, lets keep things simple. Again, water cooling adding complexity.

We are back to the starting point, do not make hot chips that require water cooling. Maybe I am too risk averse. But I worked with big computers (mainframes), then Amdahl and Hitachi came along and made air cooled machines. Which sold like hot cakes as they did not require all the extra baggage of water pipes, cooling, power, monitoring, maintenance. If you want to scare the hell out of a datacenter manager, tell him he has a water leak in the datacenter.

Prevention is better than cure, do not use water cooling.

I am not that old, but strive for simpler computer architectures avoiding the mistakes of the past. Have we learnt nothing from our computer heritage.

Water cooling

I'm with the pro water cooling crowd.

Water cooling might be a bind if you are thinking in terms of a small server cupboard cooled by through the wall air-con units, but once you have a whole building full of the beasts then your air-con system has to be driven by a massive chilled water plant anyway.

In this case getting the chilled water closer to the source of the heat has to reduce the overall power use.

The only time I remember a big puddle of water under the machine room tiles it was not a failure of the chilled water system but the dehumidifier drain pipe got blocked and it was the air-con system that caused the flood.

Sooooooooooo

We struggled with the Stone Age,

Battered our way through the Bronze Age

Ground through the Iron Age,

Had a bit of fun with the Industrial Revolution

Entered the Space Age

Landed in the Garb Age

And are now having a second go with a renaissance of the Steam Age

@Valdis Filks

Hitachi and Amdahl pushed air-cooled solutions because they were competing with IBM on price, whereas IBM was compteting on having better performance. IBM knew that Emitter-Coupled Logic (ECL) was a faster clocking solution than CMOS or NMOS at the time, and that despite the heat they would have a performance advantage on the very heaviest workloads.

Eventually, CMOS and NMOS gained in clock speeds, and even IBM made the transition - but acknowledged that the individual processors were slower than their predecessors. Today's energy-efficient CMOS processors require massive paralleization to compete with either more complex CMOS designs (i.e., the original scorching Pentium IV, et al) or ETL successors.

And what has the parallization of processors brought? An acknowledgement that for many sorts of problems, we can't effectively write parallel code. That's why Intel and others are crash-researching parallel coding techniques. That is why IBM continues to push the speed of individual processors, even to the point of resucitating water-cooling in the data center.

Now, that's not to say that MANY problems don't decompose well into parallel solutions: parallel databases have revolutionised data warehousing, clusters are wonders of mainstream application hosting, etc. But for several, important, and PROFITABLE types of workloads (esp. some scientific work and mathematical work, as well as some database solutions) individual processor performance are the limiting factor.

Lastly, I would be curious to see how the water cooling on these corresponds to the watercooling in old mainframes. Are they running the pipes within the case to radiator fans (as in a watercooled PC), or running them external to the case via datacenter cooling lines as in the old mainframes? In short, what is the real level of complexity?

Parallelism and water cooling.

The water cooling is a risk assesment, reduce the components involved and simplicity always is a better option.

For example you can have, in your infrastructure:

Design A = Power (electricity), Network (electricity), SAN (optics mainly), cooling (air)

Design B= Power (electricity), Network (electricity), SAN (optics) and cooling (air + water)

Design B has more components plus a catastrophic mixture of water and electricity. This is the problem, even if you use it to heat your building, your infrastructure just got very complex and dangerous, by adding water. Your safety regulations and such like has just increased costs.

Parallelism, this is also a virtualisation play.

With a low power air cooled server which has 8 cores you can virtualise 8 single threaded apps. You use the free software and the system to create 8 domains. As some of these cores can have more than one thread e.g. 2. You could even consolidate 16 servers or to one of these highly multithreaded servers which are 2U in size and go by the name of coolthreads. No water cooling required.

However, lets take any mainstrean already existing database e.g. MySQL, Oracle, Postgress and DB2.

These have many parallel processes (often more than 8) e.g. db writer, lock manager, transaction manager and so on. So put these on a multi core/thread server and you have a good match. No coding/changes/migration required. A very large majority of servers are running OLTP workloads that have a database engine as described above so a very large segment of the computer/server market is ideal for parallel computing.

Long term agree, need to write better parallel apps. But a lot of apps (OLTP) out there are parallel in nature already.

Can't quite agree on the water cooling, but I am open to suggestions.

Flawed thinking

Re: "Water cooling for CPU's is an admission of a design failure. The extra pipework and electrical power to run the water cooling just adds costs and complexity. Extra electicity is required to run all the water cooling, so there are not savings, this needs to be taken into account and cannot be a hidden infrastructure cost. Plus to add an extra water infrastructure to a computer center with all that electrical wiring is dangerous. Computers and water do not mix well together. Using water cooling for computers is a technological backward step."

Totally flawed thinking. Your suggesting that it's not worthwhile using water cooling as it requires electricity which would negate the benefit caused by the cooling process itself.

Total rubbish. The idea behind water cooling is to cool the hot processor, to allow that processor to run at a high clock frequency. Who cares if the cooling system requires some electricity to run it, the goal is not to save electricity, the goal is not to prevent the heat being generated, the goal is it to allow the processor to run at 5GHz to take that heat away from the chip and if water cooling is the way to do it then that's the way to do it, if forced air cooling is not sufficient.

What do you suggest? Not use water cooling, if so, what do you then do? The chip will burn out and be completely useless.

Think about automobiles, in most the engines are water cooled, and yes, it requires electricity to pump the water round.

By your argument, nearly every automobile that has ever been made is a design failure which is clearly nonsense.

Years ago, mainframe computers used ECL (silicon based chips) which drew considerable more current than current CMOS/MOS technologies, they were water cooled. If properly designed then it's not a problem.

Water cooling adds complexity.

Agree, water is used in chillers in the datacenter, most of the computer rooms that I have been in, the air con units/chillers are in the periphery/edges of the rooms. What water cooling to servers/computers does is put pipes and plumbing all over/under the floor mixing it with the network, SAN and electrical power. Do we need an extra substance/piping under the floor. Does it create more problems than it solves. I have seen many companies go through projects to simplify and reduce their underfloor/overhead wiring. Some datacenters even have alarmed floor tiles. Why do you want to mix water pipes with this.

The point of this is to reduce complexity, less is more, simplicity is better than complexity.

All computer manufacturers should help give the computer industry a good reputation for technology leadership by reducing complexity. This is the point about water cooling, it does not make things simpler for customers. Water cooling makes computing more complex.

Do you want to say to your customer, here is a new server and do you know a good plumber, on 24hr call out, 7 days a week.

NB I do not write anonymous but am open and stand behind my beliefs. I was born in a country and live in a country where free speech is respected. Do anonymous writers/comments have something to hide ?

Air cooling aint that great...

back in the IBM water cooled mainframe days it was the AIR cooling devices that were failing, not the WATER cooling. Dead fans caused more outages than anything else. So now there's redundant fans all over the place...not exactly simple...or quiet...or efficient. How many blade racks have caused hot spots in computer rooms due to expelling too much hot air?

I can't help but think

In these days of high energy costs, and growing environmental concerns, that water will win. You can dump heat from a water system by pumping it through a pipe in the ground. You could easily heat exchange it into an existing HW system. With a bit of forethought even the worst plumbing failures could be manageable - with reduced capacity.

More watercooling opinions

The critique of watercooling as I see it has a main flaw. While much of the reasoning in the examples is ok and appropriate the generalisation does not follow. So while it may very well be the case that there are questionable implementations of watercooling this is not a natural result of the principle of using watercooling. As a complementary proposition it is perfectly feasable to imagine flawed aircooling solutions. As I have come across quite a few I do not even have to use much imagination for this. And it is true that we could find many flawed watercooling systems that also fail. It is also true that we could find systems where watercooling was put as an afterthought because the original objectives of the design were not met. However this does not mean that it could not be possible to design intentionally a solution with watercooling (e.g. not a result of a 'design flaw'). In any specific practical implementation such a solution does not by default have to be more prone to failure than a specific aircooling solution. Compare to a car engine, while it could be argued that watercooled car engines in general add complexity (and would in principle than be more suspectible to problems) this does not necessarily mean that air cooled engines are more reliable in practice. We should also remember that if complexity is something we do not like than maybe we should stay away from all things IT - what about the result of moores law?...

One thread, one processor

Overclocked gaming PCs use water cooling, but not in the same fashion as the mainframes of old. So I think that the risk of leaks can be avoided. Since the real need for high speeds is for problems that are serial, I would think that a limited number of hot, high-speed chips would be needed, and the parallelizable part could be taken care of by slower multi-core chips that produce more throughput per watt. It would be nice to get a desktop machine with just one of these high-speed processors, but it would have to run Linux, not Vista, not being x86.

RE: Valdis, water vs air cooling

If the machine did not use water cooling, then it would use heat pipes with gargantuan fins to disipate the heat into the air. The fins for just one CPU would be enormous. The fins for *32* CPUs would mean that the case would either have to be huge, or else take another rack space behind it for the CPU fins.

I have worked in datacenters where I had to wear ear muffs so I wouldn't be deafened by the fan noise. The datacenter at Microsoft which has two HP Superdome servers has cold air running through it like a wind tunnel. Nobody stayed in that room longer than they absolutely had to.

Now consider water cooling. Water is inexpensive, and easy to handle. Large HF transmitters are water cooled, with the tubes in direct contact with pure circulating water, and nobody gets electrocuted and the equipment runs fine. Fluorinert is horribly expensive, and water is, well, cheap as water. Yes, plumbing is a problem if your plumbers are incompetent.

From what it looks like, your position is that anything other than air cooling is a failure. Sorry, 'tain't so.

Liquid cooling?

Makes sense for situations where you've got to conduct a lot of heat away. You trade off the extra infrastructure for efficiency, quietness, even heat distribution (important) and even the potential to recycle some of that energy. Most car -- and motorcycle -- engines are liquid cooled (it may be a surprise to know that many bikes are liquid cooled, the fins on the engine are just for show).

Air cooling in data centers doesn't make sense. You make a lot of noise (and spend a lot of energy) transferring the heat to the air only to have to remove it using liquid cooling through air-conditioning. Its much better to cut out the middleman.

Water works well. I've been on a project where we used de-ionized water because we had to plumb it to a tube anode running at 7.5Kv -- taking the ions out renders the water non-conductive. For data center use, though, the water's more likely to be like the stuff in your car engine.

This vs Sun's gear

I checked out the Sun UltraSPARC T2+ today - that seems like a (really) fast processor as well (or so the benchmarks suggest).

Is there anywhere I can see the benchmarking results mentioned in the article. I just want to check out the number of CPUs in the competing products ( I have a tinkling it's going to be somewhat less than 32.)

plus, on a price to performance ratio basis, POWER6 is horrible (IMHO).

Hi Valdis

The use of air for cooling is the admission of lack of technical skills. It is the most power-hungry and inefficient solution around. It is also much more complex than water cooling, and not safe at all (you need to filter and cool humongous volumes, and the fans are very prone to failure).

"I was born in a country and live in a country where free speech is respected." I believe you never ever went to the glorious US of A then.

Benchmarks

Hear hear to the man that praises the SAP benchmark. If you want an application engine to run Java or an HPC environment then by all means look at SpecJBB, SpecFP and all those compute-intensive things, but if you want to see how a database works (and there's few other reasons why you want a big shared-memory multi-processor machine) then there aren't that many good benchmarks around. TPC'C' tends to be more of an exerciser of I/O systems than anthing else (some configurations have over 7,000 disk drives), TPC'H' is OK for data warehouses, but is not readily applicable to transactional systems.

No, if you are stuck wihy a complex ERP package because some architecture team has decided that SAP, Peoplesoft, Siebel, Amdocs CRM or whatever has to be used because it is "off the shelf" and you get faced with some huge, single instance database which doesn't readily lends itself to parallelism, then SAP is about as realistic as it gets. It ain't perfect, but it's the best of a bad job. In a world where benchmarks are marketing tools, where software and hardware vendors contractually prevent the publication of benchmakrs (including those by customers who have paid for the stuff) then there's not much else.

I've no doubt that the new IBM will be the fastest single-instance DB server out there (and we don't use AIX so I'll count myself as unbiased). However, if you do make comparisons then make sure it's the same database, same version (near enough) and don't try and compare UDB/DB2 numbers with different SQL*Server or Oracle.

Oh yes - and back to water cooling. The biggest problem with that for IBM is going to be that very few datacentres are set up for that. Unless IBM have a very slick short cut then that's going to be a problem (unless all they are doing is piping the water to a heat exchanger in the cabinet - which is not the best thing to do, but makes installation vastly easier).

IBM's POWER Platform

IBM has in fact released the POWER platform. It's like the Intel platform only faster and stronger with much more history and reliability.

The Statement: "It will ship this May with AIX and Linux" is mostly true.

The Power platform comes with nothing on it and you as the user can place a mix of AIX, Linux or i (formerly i5/OS or OS/400). The System is and has been for the longest time a virtual haven. Before VMWare was VMWare IBM was doing partitioning .

You can run a mix of OSes on it and before too long you will be able to run Intel based Linux and I would guess Intel based Windows as well.

POWER as a platform is going to change a good many things. I don't think Sun is ready for what IBM has in store.

apple

I want a P-6 Mac!!

Apple may have had a good reason to go with (ugh) Intel, but they should have kept

Power PC available for special order!

Imagine a Mac with a dozen or so of those 5GHz P-6 processors in.....

Evian for the datacenter - not good

I am still open to be convinced that water cooling is better than air. It may be better but not one post has explained the advantages yet.

Why do we make such hot chips. A hot chip implies lots of power/electricity. Then we use electricity to drive a water chiller, then we drive an electric pump. If the chip design was not this hot in the first place then we may not need all this extra cooling. This is the prevention rather than cure argument. Hence my argument why the design is flawed. It was made too hot first of all, then we have to get the plumbers in to cool it down.

In these times of global warming we should be looking for ways to make cooler chips. Can a company use a water cooled Power6 server, install new water cooling, do all the extra work and then say that all these extra resources are good for the planet. Alternatively install a low power, multi-threaded CPU/server, which uses air cooling.

If people think that Power6 is good for heating buildings, then we can use the existing air-cooling systems and the heat they take out of the datacenter to heat the buildings aswell. Just redirect the cooling mechanism.

The new Power6 has a clock rate which is 2x faster than the Power5 (approx), but the Power6 applications may not run 2x faster. The increase in GHz does not match an increase in performance. Thus why do we have such hot high clock rate CPU's. Most of the industry is moving away from high GHz hot CPU's.

The majority of companies Intel, Sun, AMD and IBM Cell make CPU's which do not need water cooling. I am a fan of IBM's cell chip, excellent design. I am not a fan of water cooling. Strangely Sun, Intel, AMD & IBM Cell are moving to more parallel/multi-threaded designs that run cool. What is P6 doing.

IBM for some reason made a variation of the P6 which required water cooling, no answer here.

Because we cool cars with water does that make it good for CPU's. We actually use oil to do the cooling aswell. Next thing which has already been hinted in this discussion is that we will have several liquids in a computer to cool it. More components to me means more complexity. With complexity comes extra costs.

Many comparisons of a new IBM server to old Sun servers, please compare new IBM servers to new Sun servers. My post was about the environment and implications of all the extra resources required to install water cooling in a datacenter. Obviously we need to look at CPU design and the hot chip GHz issues, but mainly my interest was in water cooling issues.

Computer history repeats itself, we may well see more water cooled systems. But I do not think that is a good idea for all the reasons already outlined.

I have been to the US more times than I can remember, half my family are US citizens. I am an alien (see previous comments in The Reg about me) born in a very democratic country, living in another very democratic country where free speech is very much respected. I have seen and worked in 100's of datacenters/rooms. I thought that we saw then end of water cooling, just like ECL, Bipolar Chip design.

A good plumber is hard to find.

My issue about water cooling is with the whole system, the server, the maintenance of the server, redesign of the computer room, the outages required to do all of this etc. From a thermoconductivity perspective water may remove more heat than air. But you need to get the water all the way to the server and plumb it all in. I like plumbing but this is expensive and difficult.

How about not causing the problem in the first place with cooler CPU's.

I think I mentioned and gave examples of liquids that are good for cooling and expensive implementations thereof. e.g. Magnox cooled nuclear reactors. Air is not good to cool nuclear reactors I think that I made that clear.

Cost or ripping up floors and installing pipes is more than not ripping up floors and installing pipes.

If I want to move a air cooled server I unplug the electrics, network, SAN and move it. Then reconnect.

If I want to move a water cooled server call the plumber, book a weekend, possibly shut down the whole datacenter, move the server. Lay new pipes, pressure test new pipes, reconnect electrics. Do I need this complexity extra constraints.

Water is used already in the chillers in the periphery of many computer rooms, you can already use this hot water to heat your office as a green by product. Do we want a water piping grid in addition to all the other infrastructure/cabling in a computer room. Every computer room is different and needs unique/bespoke plumbing to install water cooling to a individual server. For the sake of one hot Power6 server, am I going to re-design the whole datacenter.

If in a couple years time when the whole industry has moved to cooler multi-threaded CPU's/servers, will your investment in water pipes all over the computer room be a good thing. Or will the computer industry employ legions of plumbers to rip out the water pipes that they installed 24 months ago.

I may be totally wrong and water cooling within servers may become more popular and economical, at the moment I find it difficult to believe.

more power Igor

Watercooling for servers is never going to be popular or economical, for all the reasons listed. Water and volts don't mix, even slight accidents are catastrophes. Anything other than HPC and specialist sites will run a mile from the hassle.

IBM don't say how much power these beasts consume and how much heat it puts out. Kilowatts/Hour and BTUs/Hour please (even my electrisave can calculate Kg/Hour of CO2) and then we'll decide if its "green" or not. In the current climate (ouch), these figures are as important as how many tedious SAP users it can support. Power is power and heat is heat, quite how you shift it from the chips to outside the datacenter is moot. Sun get this, I think IBM are too busy outsourcing and consulting to care anymore.

Need to solve both sides of the equation

Valdis is right when he says we have to reduce the power requirements of CPUs, etc., but his dismissal of water cooling is like preventing roads from being built because someone may actually want to drive on them.

Sun tries to solve for both sides of the equation with lower energy servers and cooling technologies. Sun has the CoolThreads CPUS (Niagara & Victoria Falls), but also the Sun Modular Datacenter (widely known as Project Blackbox), which utilizes water cooling and efficient packaging to reduce cooling costs by over 40%. This savings is independent of payload type or vendor. However, if you load the Sun MD with energy efficient systems, like the Niagara servers, the savings are magnified.

hmm nice to see

that this didn't turn into a bun away topic do to my typo.

Yes that's my straight jacket then one that smells like suggar

Will we have an iCooler next ?

The SunMD is a standard design with water cooling built in the factory, every SunMD is the same. No need for any local plumbers to change anything inside when a customer receives it. Just plug it into the power and external cooling pipes. When you buy a Sun Modular Datacenter (aka Blackbox), you do not not have to change anything. Put a water cooled server in a computer room and as explained many times, you need to do lots of extra work and ongoing complexity.

As far as I know all servers in a SunMD are air cooled, no servers are water cooled. But I am sure if someone paid Sun enough money we would be able to connect a water cooled P6 in the SunMD to the pipes. I would advise against this, guess why, it adds complexity.

SunMD is water cooling outside of the computer chassis/enclosure. The Power6 water cooled server is water cooling inside the chassis/enclosure.

My point is about the added complexity, which water causes if you have to put it into an existing datacenter.

Now if people want to make hot chips or more elegant designs then we technologists have a challenge. Produce a coolant that is safer to mix with electrical devices and those nice little towers that we put on CPUs can become a selling point, lets call it the iCooler. I remember that some of the IBM, Hitachi or Amdahl mainframes had those elegant circular tower heatsinks.

Well it was pretty but complex. Now maybe many overclockers like to cool their PC's at home with these type of things. The modders always like new gadgets and to spend time tweaking their systems. Commercial datacenters do not.

NB I built my latest PC with the criteria of least power usage, it is based on a AMD BE-2540 dual core cpu, $ per performance per power usage it was the most efficient. It may not be the fastest, but I was being Mr Sensible. No water anywhere near it.

In commercial datacenters I do not think we can overclock, mod and customize our servers with lights, water coolant towers etc. But maybe the first person to do so could make the datacenter into a work of art and a light show.

Me no iModder.