Change the form factor

Many clusters are built of 1U multiprocessor pizza boxes with 2 or 4 CPU + a PSU per box.

Could these things not be offered with the power drawn from per-rack bus rails? If so then perhaps a single large PSU for each rack could be engineered for better efficiency.

Instead of a full 110/220 VAC PSU in each pizza box, just have an essentially empty PSU that plugs into the power bus - the rest of the pizza box can be the same product sold for non-clusters.

Also the comment above is good: build data centres where there is a lot of cold water already. Canada has good power infrastructure and a lot of cold water.

Good ideas

"...the Blue Eaters machine is supposed to have more than 800 TB of main memory (with at least 32 GB of main memory per SMP node), and more than 800 TB of main memory."

I believe that statement is repetitive and redundant.

"...comprised of 162 racks of servers, organized into three columns, with each column having five rows with 9 racks each."

3 columns times five rows times 9 racks = 135 racks. Where do the other 27 racks go?

"The Blue Waters data center will not have room-level air conditioning, either..."

It's not my place to question data center designers (having no personal experience with data centers myself), but from a layman's viewpoint, won't the data center still get hot? Even if you can pipe all the heat away from the processors and memory using the water cooling, what about the hard drives? Will those also having water-cooling blocks attached to them? If not, won't the spinning platters and moving heads generate a good amount of heat?

At any rate, it's good to see someone thinking about data center design to increase efficiency.

Re: Lots of cheap electricty and cold air up north

So instead of combined heat and power systems, we'd be taking combined heat and petaflop systems. I like it

@Good ideas

It's actually going to prove quite a challenge to cool the drives.

The water spins straight off them (ducks). :-)

On the serious side, I think there will be some more ideas before this goes live. Motherboards, especially caps, require a decently controlled temperature too. That ought to be easier to manage, though, if you deal with the main source of heat first.

I've had this idea of a watercooled rack for some time, but I've cut down on hardware recently so I don't think I will bother. Anyone want a half height rack + shelf? :-)

Is this correct?

"..a 300 watt device running in a data center requires 800 watts of input power. .."

I can readily believe this, what with the inefficiencies at each stage of the power supply chain and the large burden of providing power for the cooling systems. By 'input power', I assume this means total power supplied to the entire facility from the electricity provider.

The final statement of planned and calculated savings seems way too optimistic however:

"So to power up a 300-watt device in this data center will only take about 350 watts of input power. That's a very big improvement."

That's not a 'very big improvement', it's an unbelievably enormous improvement. I would ask if this compares like with like. The initial figure of 800W to run a 300W box referred to total power provided to the facility, with all ancilliary equipment (power chain, cooling, etc).

If it is the case that this level of improvement can be achieved with simple design and usage techniques, then why on earth hasn't it been done before in other data centres, resulting in massive running cost savings (as well as saving the planet etc). ?

It's probably too much to ask but I'd like to see a follow up study done on this centre when it's completed and in normal operation. That would be interesting.

Best Heat Sink

Is, of course, the oceans. Pick a location where the sea is cold all the year round and use a heat exchanger to dump the heat. That's something the designers of Nuclear power stations have known about for years. Air cooling is never as reliable - even up a mountain (at least to accessible heights), the air can get warm. At high altitudes the air density is significantly lower which lowerstthe effectiveness of heat exchangers.

Of course you want to choose a place where the sea is cold all the year round - but the USA has Alaska, and the sea there is very cold (plus I assume they have significant hydro potential). Even further South the Pacific is generally pretty cold as there's not much of a continental shelf (although being on a subduction zone, then earthquakes and tsaunami are always a danger).

Of course being posted to Alaska might not be that popular....

The question on everyone's lips...

Will it run Vista?

@ Chris C

The storage boxes attached to the clusters can be in a seperate room if necessary. The nodes themselves would not need any hard disks. Note that the storage boxes can put out a fair bit of heat but even so that would only be a fraction of the heat put out by a high density rack.

Most current high density racks blow out an incredible amount of heat (standing behind a rack of HP bladecentres is like you just opened the door of your car on a hot day), but almost all of that is from the CPUs, PSUs and chipset/RAM. The remainder of the circuitry generally does not even have heatsinks on them. If you cool the ram/chipset and CPU with water and the power comes from somewhere else then there will be very little heat generated by the rack and minimal cooling would be required.

Ideally as mentioned these datacentres should be put in places with cheap power and low temperatures. Also a good idea mentioned above, the hot water produced could be piped to nearby residences and offices during winter...

@Gordon Henderson

Thanks for explaining!