back to article IBM shows off Power7 HPC monster

IBM likes to go on and on about the transaction processing power and I/O bandwidth of its System z mainframes, but now there is a new and much bigger kid on the block. Its name is the Power Systems IH supercomputing node, based on the company's forthcoming Power7 processors and a new homegrown switching system that blends …


This topic is closed for new posts.
  1. Simon Casey

    So why buy a Power6

    So why buy a Power6 box with this announcement coming out? At least in the mid to large range size of boxes.

    Ahhh the return of watercooling. I assume AIX will be first on this platform, followed by probably OS/400 and then ... z/OS?

  2. Peter Gathercole Silver badge

    Lead time and who-wants-to-buy-Power6

    The lead time for installing this type of system runs into many months, because of the associated infrastructure required to put this type of system on the floor. This means that most of the customers who might want one on launch date will have to start planning their power, water cooling and reinforced suspended floor now in order to get one installed close to GA date. All of the big Power6 build capability IBM has is probably already committed, so a customer probably can't by a Power6 HPC now even if they wanted one.

    Power6 575's are already water cooled. The IBM regional hardware CE's have to do a water cooling training course before they are allowed to touch the systems. The smaller systems are air cooled, and the smaller Power7 systems will also probably be air cooled as well.

    And you have to remember that there are only a hand-full of customers in the UK who have pockets deep enough and problems large enough to warrant purchasing this model of Power7 systems (see the top500 to see who they may be).

    As far as I am aware, although i (aka i/OS or OS/400) will run on these systems, it is unlikely that the applications that will run will be able to take advantage of the capabilities of one of these systems. It is unlikely that anybody will bother to LPAR them, even though it may be possible (it is certainly possible to LPAR a p6-575). Looking at p6-575's, they are every bit CHRP POWER systems, with similar entitlements and capabilities, and even the same basic AIX installs. From a system management perspective, they are just very large AIX systems.

    The current zSeries systems run significantly different silicon. Although the fundamental processors are similar in design and structure, the actual instruction set, even allowing for micro-coding, is too different between Power and zSeries to allow one to run the other's programs.

    Recent zSeries systems have also been air rather than water cooled. Mainframes excel in huge I/O performance rather than just basic grunt.

    Of course, Linux is always an option.

  3. spoonwzd

    Yes, yes, yes...

    ....but how many FPS in Crysis??

  4. Simon Casey

    The Power Equation

    Oh I don't know. If you're properly organised you can get a 595 up and built and running within 48 hours with everything prepared for it to arrive in a datacentre. Even less if you've got enough manpower. And the shops that need this sort of clout if they really push for it will get it before GA I would guess.

    I'm just wondering how many large shops will really want to commit to buying a 570/595 sized box on Power6 and not instead at least wait until they can see feeds and speeds for Power7 and it's prices and when it'll GA. Heck some of the Power6 boxes really have to have very specific applications to take advantage of all their CPU power as many will be storage or memory constrained. Some of the 570/575 boxes memory per core count is a bit strange sometimes.

    I wasn't considering i/OS or AIX as a single partition on this sort of setup, yes for HPC but no for general commercial applications. I was wondering about systems this size for a purely consolidation of a lot of lpars into one footprint, albeit you'd want another footprint elsewhere for HA or recovery.

    I'm glad to see the switching architecture is meaty enough to start giving SAN attached infrastructure a run for it's money. Much of the Power5 and Power6 kit will not drive 4/8Gb SAN equipment to it's true limits even if properly tuned in a commercial environment. At least it's finally shifting the bottleneck off the host with this sort of equipment. I hope that there'll be some sort of performance analysis tooling with the switching kit in these boxes though as working out HSL bottlenecks is of course a little difficult.

    Yes I agree the zSeries does have a different processor architecture but I do not think that is the current reason why IBM have not transitioned the zSeries to Power processors. If you look at zSeries the biggest stumbling blocks are the I/O systems that z/OS is built on utilisation HCD and cascaded ficon. I would bet that IBM can already easily run z/OS on Power but it's just shops being able to port the other attached infrastructure over to it. Just like 6r1 from 5r4 yes there were little tweaks even with the TIMI but it's not a huge pain. A fundamental I/O system change would be though. I wonder myself (personal opinion) if z10 will be the last non-Power based Z box.

    Ahh didn't mention Linux because once IBM generally have AIX running on it Linux would shortly follow. I bet availability will be AIX -> Linux -> i/OS -> z/OS if at all.

    The global address space scheme really does sound interesting, guess it's like the ideas in EMC's v-MAX.

    I take it you're not the Peter Gathercole of Fly-fishing fame? But probably the ex-ICL man? I grew up on a set of SY and SX ICL boxes and shockingly I'm under 30!

  5. Destroy All Monsters Silver badge

    Wrong question....

    "but will it run Cat Brain" is the correct one

  6. Robert Hill

    Sure it's got water cooling...

    So it's got water cooling, big deal. How many blue cold cathode tubes does it have (they'd have to be blue, wouldn't they?)???

    And spoonwzd, we need a new metric - how many fully interactive, 3D-display using, Crysis users can it support? I bet a fair few...

  7. Anonymous Coward

    Re: As/400

    "AS/400 designs from 1995 (not the crappy PowerPC chips that came out of the IBM-Motorola-Apple partnership)" ---- I recall the "crappy PPC" blowing the doors off of the AS/400 which was well into legacy retirement by that time anyway.

  8. SpinMe


    i miss the point. will it sit at 10% cpu load ?

    If you need one of these servers (and your not doing research) then your design is hugely wrong.

    Honestly, most servers do bugger all for 80% of the day. If you utilised the spare capacity instead of trying to make a black hole on earth then powergen/eon would be out of business by now (that's a good thing).

  9. Robert Hill


    It is NOT a file server, it is an application server, for applications that can take days or weeks to run to completion. Such iron (and I have built a few of these) requires throughput that will stay pegged at 100 percent for long stretches of time - like the Orion demand planning SP2 (with terabytes of storage and tens of CPU nodes back when they were still expensive as hell) I helped install at United Airlines a decade or so ago - it ran at 100 percent throughput for 18 hours of every day, with checkpoints, to ensure that it could be restarted and still be able to run the statistical model to completion at least once per day. This ensured that seat prices were updated once per day to maximize revenue. And that was an EASY commercial application compared to the immense stuff some of these classified servers run - nuclear simulations, weather prediction, etc.

    Believe me, these beasts do not sit idle for very long - too expensive to let go to waste. Usually there is a long waiting list to get time on them...

  10. Kebabbert

    Holey Moley

    800 Watt?? Are you kidding? Earlier I asked if the Power6 consumed 500 Watt. But Jesper Friedman told me I was totally off. I wonder how much off, I was? I wonder if the Power7 and Power6 consumes almost same amount of Watt? They decreased the Hz on the Power7, so the cores seems to be slower than a Power6 core. Instead you have 8 of the slower cores. This reminds me of something... Hmm... Use more cores, but slower.... Hmmm... No I cant recall. But I know IBM and you guys here, downtalked CPUs with many slower cores. Few fast cores is how you do it, IBM told us. So, where is the new IBM dual core CPU with 7GHz? Or has IBM changed direction to follow other chip manufacturers who use many, but slower cores?

    But the stated 1 TFlops for 8 cores is impressive. I would be more impressed if IBM did it using less wattage. The problem is not build a fast CPU. The problem is to build a fast CPU abiding constraints. If you have unlimited money and unlimited power and unlimited everything - then you can build a fast cpu. That is not the problem. The beautiful thing is to build a fast cpu under heavy constraints. To make beautiful solutions. If you are not trying to be clever, you just throw more resources and money and power at it, instead. That is not sophisticated solution, but a naive solution. I wonder how this CPU fares in a mainframe. As I have shown earlier, the mainframe CPUs are really slow and sucks, as 1MIPS == 4MHz x86. IBM states this CPU does 1TFlops. I want to see benches proving that. Or is IBM lying, as usual?

    I would like to see the benches on this CPU. Then maybe I will say, just like the rest of you, "no, it is IBM that benched that, so you can not trust the benches. The real numbers are totally different. This is a cherry picked benchmark and not to be trusted". Just as you guys say, when I post benches for Niagara.

    A question: earlier IBM stated the Power6 has 250GB/sec bandwidth. Now IBM states the Power6 has 100GB/sec bandwidth? Or was it Power7? So what is the true bandwidth? 250GB/sec as earlier claimed? Or is it 100GB/sec?

  11. Kebabbert

    800W for a package of 4 cpus?

    After rereading the text it seems that 800W is for a package of four Power7 cpus? Ok, that is much better. 200W per Power7 is actually acceptable. Then the Power6 is maybe in the same vicinity, around 200-250Watt. Which shows that my question was off with a factor 2. Not 500W. Which shows that Jesper Friedman was right, and my question was wrong.


    1Tflops for 4 Power7, equals 250GFlops/cpu. This is twice as fast as Fujitsus Venus Sparc CPU which is reported at 128GFlops. However, the Venus consumes power "a third of what Intel Chip consumes"

    If Intel Chip consumes 150W, then one eight core Venus consumes 50Watt. This is only 25% of Power7 power consumption.

    And the Venus is also lower clocked. If Venus uses 2GHz vs Power7 uses 4GHz, we see that if Venus is clocked to 4GHz, Venus will produce equally many flop, clock for clock as the Power7. Venus will then produce 256Gflops, just like the Power7. But the Venus consumes much less power. Hence, the Sparc is at least twice as efficient as Power, probably a lot more.


    Because at high clock speeds, you depend upon the work load fit to into the cache. Otherwise you are smoked. The more Hz, the more you are punished when your data is not in the cache because there will be large differences between CPU speed and RAM speed. If a CPU runs at 10GHz, and if RAM runs at normal 1GHz, the cpu will wait 9GHz for data from RAM. This means 10% utilization. Therefore, it is better to have lower clocked, because then you will not be punished as much by cache misses, and you will have much better utilization. It is dumb to try to ramp up the Hz, which IBM does. The cache misses are far too penalizing. Better to use a cpu which is equally clocked as RAM, when we talk about utilization and power consumption. If you have bad utiilization, you have to use maybe 200W or so to do work which a lower clocked CPU can do with better utilization and efficiency. The Power architecture sucks at utilization and efficiency, because it relies on high clocks.

    Of course you could change direction for Sparc and throw more power at a SPARC chip, but that is not wise. There is a border where you can optimize performance vs power. SPARC is highly optimized for that. IBM optimizes highly for performance, that is all they care about. Which is evidenced by, for instance TPC-C. That is just a matter of getting high numbers. Dont matters how much resources are used. That fits IBM well: "No constraints". But there is a larger challenge to abide to constraints and still win.


    Power consumption is a major constraint and hurdle in large supercomputers. IBM was dumb to put four Power7 into a package which consumes 800W. Compare that to rank no 6 today in Top500. Rank no 6 is Blue Gene, which uses 800MHz PowerPC. How much power does this 800MHz CPU use? 50Watt? This shows it is possible to achieve high performance without using much power. I wonder if there will be large supercomputers using the Power7 package, because you would need a nuclear plant to drive that thing. Better to use lower power consuming CPUs in large supercomputers. That is why I doubt we will see large clusters of Power7.


    However, IBM stated that the Power6 has 250GB/Sec bandwidth. And now, in this article it seems that Power7 has 100GB/sec bandwidth? Has the Power7 lower bandwidth than the Power6? Why is that?

  12. sT0rNG b4R3 duRiD


    Ok, now tell us what you're doing for the PS4, now that I hear the Cell's been killed?

  13. Toshiro

    Out of juice?

    Sounds like a step backward for IBM.

  14. Chavdar Ivanov

    Imagine a Beowulf cluster of these...

    Oh, wait, this was not /. . . .

  15. Anonymous Coward
    Anonymous Coward

    10Gbit/s commodity ethernet switch

    Brenner said that the hub/switch module delivered a 1,128 GB/sec - that's bytes, not bits - in aggregate bandwidth.

    What is special about that? That's a commodity switch running ethernet IEEE 802.3ae with 10GBit/s.

  16. henriok

    1000 port 10 Gbit-switch?

    Yeah that's a commodity 10 Gbps switch with 1000 ports. Where can a buy one of those?

  17. tom 99


    Kebabbert, you need to educate a bit more before posting such a mess.

    Clock frequency has little in common with CPU performance in this case. Single Power7 core is faster than single Power6 core, because P7 has out of order processing.

    Other assumptions of yours are wrong as well. Reading Sun whitepapers is not good enough.

  18. Anonymous Coward

    correction please

    "The hub/switch at the heart of the Power7 IH node and linking them together is the secret sauce of this machine. Brenner would not elaborate much on this network, but did confirm that it borrows ideas from the "Federation" SP hub/switch IBM created. This was for ASCI Blue and other supers running AIX and InfiniBand switches and related InfiniBand technologies Big Blue has been using to link Power5 and Power6 processors to remote I/O drawers for years."

    Federation was the interconnect for ASCI Purple.

  19. fdsa

    Re: 10Gbit/s commodity ethernet switch

    "Brenner said that the hub/switch module delivered a 1,128 GB/sec - that's bytes, not bits - in aggregate bandwidth."

    "What is special about that? That's a commodity switch running ethernet IEEE 802.3ae with 10GBit/s."

    What are you talking about? You'd need about 900 10 gig links to get 1,128 GB/sec of bandwidth.

  20. Anonymous Coward

    Re: Holey Moley

    "As I have shown earlier, the mainframe CPUs are really slow and sucks, as 1MIPS == 4MHz x86."

    Kebabbert, you haven't shown anything. You just repeat hyperbole from others. Why don't you come back with 2009 numbers. No one cares about z990 vs pentium 4 from 2003 anymore.

  21. Anonymous Coward
    Anonymous Coward


    The POWER chips were first used in RS/6000 AIX servers in 1990 or so, not the AS/400 in 1995.

    I guess Peter Gathercole is not the fly-fishing guy, but one of the first and leading members of the IBM AIX support team in the UK. Not sure where he works now, Deutshe Bank still?

  22. Simon Casey

    Fish n chips

    Wahey someone who knows their history :)

    You Sir are indeed correct, as/400 made the transition later (somewhere in V4rx?) and had the TIMI recompilation lark.

    Didn't know he was at the forefront of AIX. Using google skills he appears to be a consultant somewhere and was an ex-ICL man (are they still doing mainframes under Fujitsu?), didn't know he was an IBMer!

    Still, I know as much about AIX as I do about fly-fishing. Well maybe a little more than fly-fishing.

  23. Kebabbert

    Too powerhungry. Legacy techniques

    tom 99

    "Kebabbert, you need to educate a bit more before posting such a mess." ....I have double Masters, one in math and one in comp sci, algo theory. What is your education?

    "Clock frequency has little in common with CPU performance in this case. Single Power7 core is faster than single Power6 core, because P7 has out of order processing." ...Wow, isnt it what I have been saying all the time?! That Clock frequency has little to do with performance! But stupid IBM frantically claims that clock frequency is everything, and that 1.4GHz Niagara is slow compared to 4.7GHz Power6. OTOH SUN claims that clock speed has nothing to do with performance, and for that they get bashed by IBM, all the time. I am glad that you and me share the same point of view on this, and that IBM are wrong when they equate clock speed with performance. A counter proof is Niagara.

    "Other assumptions of yours are wrong as well. Reading Sun whitepapers is not good enough." ... As far I can tell, nothing in your post holds for closer scrutiny. I am most probably more educated than you. And we both agree that clock speed does not equal performance, contrary to what IBM states. So what do you mean with "other assumptions are wrong as well"? Nothing is wrong in my post. Could elaborate a bit more, instead of FUDing?


    Anonysmous Coward, 30Nov 04:52

    "Kebabbert, you haven't shown anything. You just repeat hyperbole from others. Why don't you come back with 2009 numbers. No one cares about z990 vs pentium 4 from 2003 anymore." ...What have I not shown? I can post this link again from year 2003, just for you

    Here we see that an Linux expert thinks that Mainframe 1MIPS equals 4MHz x86. So 1000MIPS equals 4GHz x86, according to him. If he lies, please point out his lies.

    Regarding "updated numbers", yes I totally agree with you. As I had said earlier, I expect a fast Nehalem to be compete with any Mainframe CPU. We see that 1MIPS == 4MHz x86. That is for Pentium 4, which is dog slow. Now, core2duo and Nehalem is much more efficient clock for clock. I expect a 2GHz Nehalem to totally crush a P4@2GHz. One 1MHz x86 has changed a lot since year 2003. One MHz is maybe 5-10 faster, today.

    OTOH, the IBM MIPS hasnt changed at all. 1MIPS from year 2003, compares to 1MIPS today. Mainframe CPUs are faster today. Instead of 10.000MIPS, they maybe do 50.000MIPS today. But it is the same metric.

    This means that the numbers from year 2003, "1MIPS == 4MHz" must be modified. One MHz is 5-10 faster today, let us be generous to IBM and say only 4 times faster. Then the new numbers will be "1MIPS == 1MHz". As the coming Nehalem with 8 cores at 3GHz will correspond to 8 cores x 3GHz = 24GHz we see that it corresponds to 24.000Mips. Get a quad socket mobo, and you will have 4 such Nehalems giving the equivalent of 96.000Mips. How much does a Mainframe giving 100.000MIPS cost? 10 Million USD? Compare that to a quad socket Nehalem PC. Maybe 5.000 USD? Buy a couple of those PC and you have good redundancy too.

    So I totally agree with you. Those numbers need to be updated to show the new x86 CPUs. They are many times faster than the old P4.


    Lastly, power7 sucks if you look at how much work it does per watt. Legacy constructions are like that: high Hz, small cache. Even the legacy constructed Fujitsu SPARC Venus is twice as efficient than a Power7. With all IBMs resources, they couldnt do anything better than this. Just like Microsoft: Windows sucks. Catastrophic.

  24. uarch


    In your rush to bad mouth the competition you've managed to confuse yourself w.r.t. the technical details. Please reread the article a bit more carefully.

    If you look closely you'll see the 100GBps bandwidth you're quoting is only the memory bandwidth per chip. You're forgetting about I/O and the SMP network.

    Take a moment and Google this year's Hot Chips presentations. According to what was presented, bandwidth for each Power7 chip:

    Memory: 100 GBps

    I/O: 50 GBps

    SMP: 360 GBps

    Total Bandwidth for a single Power7 chip: 590 GBps

    Your posts contain several additional technical mistakes/misunderstandings but enough people have already jumped on your case over it, I won't pile on.

  25. Kebabbert


    In your rush to bad mouth my post youve managed to confuse yourself w.r.t to my text. I can explain that again, just for you. I hope others dont have the same problems understanding my text, as you seem to have. And, please dont spread IBM's lies, ok?

    When IBM states that the Power6 has ~250GB/sec bandwidth, it is clearly wrong as I have showed. I will explain that again just for you. IBM are lying or ignorant about CPUs. Here goes. Again.


    Got that? If there is a bottle neck on 10GB/sec, then the chip will never be faster than 10GB/sec! That is easy to see, if you have studied higher math. There is a famous theorem in Discrete math, that says "Max Flow == Min Cut". Very sloppily, it says that the maximum flow (bandwidth) achievable, equals the bottle neck. Modulo all details.That is clearly obviously true. Just think over it a while, and try to apply common sense. Or study the proof. Here is more information on this theorem. So please, dont go and spread lies about Power7 reaches 590GBps, because that is simply not true. If you have not studied higher math, then just trust me. Or ask a mathematician on this.

    Clearly, you have not understood anything of that I write, because you claim that I am wrong. Study some math, and then reread my post?

    Regarding bad mouthing the competition, clearly you have not been here for too long. Do you know what lies the IBMers say about SPARC Niagara??? That pisses me off. I dont mind if IBM's Power is faster, but I hate lies and FUD. If they can show that Power is faster than Niagara with benchmarks, I am quiet. I will not say anything. But when benches clearly shows that Niagara is faster in many different aspects, and STILL they lie about it, that pisses me off.

    Fair should be fair, and no lies, eh? Lies pisses me off. So, please, dont spread lies about Power7's bandwidth. As long as they continue to spread FUD and lies here, I will do my utmost to dispel the FUD and post benches showing the truth. For instance, the lies about Power7's bandwidth. Or the lies about Mainframe CPUs, when they probably loose to a fast Nehalem. Or the lies about Mainframe is able to consolidate 1500 x86 servers, when the x86 servers are in fact, idling. etc. etc. etc etc These lies pisses me off. If the lies were true, I would shut up. I mean it.

  26. uarch



    Calm down. You're perfectly aware that's how they report the number to the press. You and I may not like it but that's the way they do it. I made a point of providing the breakdown of SMP, Mem, etc bandwidths because of this.

    If you look back at your own comments you'll see that I was responding specifically to this comment of yours:

    "However, IBM stated that the Power6 has 250GB/Sec bandwidth. And now, in this article it seems that Power7 has 100GB/sec bandwidth? Has the Power7 lower bandwidth than the Power6? Why is that?"

    You, yourself, chose to compare the total aggregate Power6 bandwidth to only the memory bandwidth on Power7 and claimed the bandwidth from Power6 to Power7 was reduced. If you want to use the 250 GB/Sec figure for Power6 then you need to compare it to the 590 GB/Sec figure of Power7. If you want to use the 100 GB/Sec memory bandwidth for Power7 then you'll need to compare it to the same memory bandwidth figure for Power6 (you can Google it if you like, I kept finding the peak figure). You could also compare the peak memory bandwidths though those are less meaning full (50GB/sec for Power6 vs 180GB/sec for Power7 - according to a recent CNET article).

    Whatever you do, you need to do an apples to apples comparison. Your previous comment compared an apple to the entire produce section.

    "[...] That pisses me off. I dont mind if IBM's Power is faster, but I hate lies and FUD. [...]"

    I understand your frustration but your passion for SUN's products is giving you a bad case of tunnel vision. Spreading even more FUD is not going to accomplish much. Both companies make good products and neither one is the perfect solution in every case. One is the right answer for some problems, the other is the right answer for another set of problems, some problems can be solved by both and some can't be solved well by either. Blindly screaming "SUN GOOD, IBM BAD" while misquoting figures and misinterpreting values accomplishes nothing.

  27. Anonymous Coward

    And you have to remember that there are only a hand-full of customers in the UK


This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2018