Re: Hang them by their lab. coats.
"I suppose that normal commercial systems vs. HPC systems is a bit like the difference between a Ford Transit and a Formula 1 car"
Yes indeed. The F1 cannot be built by strapping a bunch of Transit engines together; nor can you take the ubiquitous "white van man" and plant him/her into an F1 car and expect anything like a good result.
Or can we? We do have drivers moving "seamlessly" from a Dacia Sandero into a Bugatti Veyron. It's the Engineering of the latter that facilitates the transition. It's a vastly more sophisticated car yet is as simple to drive as an ordinary hatchback; even at much higher speeds — though not necessarily the top 10% of the performance envelope.
To carry the theme further; we have a road traffic environment with mostly much more powerful, faster cars than ever before and the driver are becoming less skilled and interested; yet the roads are far safer, cars are more efficient and (relatively) seldom need to be caressed by a magic monk in blue overalls. That "miracle" is the result of Engineering over the past 60 years.
In my initial posting, I focused on the inability/unwillingness of the industry to provide a platform in which those who are not au fait with the particular intricacies of a specific machine can still extract maximum performance. That's like Bugatti selling the Veyron and insisting that the be driven by Bugatti's own chauffeurs.
I don't make that comparison out of ignorance. I make it to highlight the shortfall in real Engineering in I.T.. Sure; the make the lights blink faster and the bits bang harder; but most don't seem to have tried to fill the need for machines that are truly useful to the people who can make productive use of the machines. The people who are the interface to the real world.
It's as absurd to expect application "programmers" to write code differently for HPC than for a "conventional" environment. It's a distraction from the problems that they *should* be solving, is not a good use of their time and more likely than not, will be done in a half-@rsed way; requiring it to be redone, over and over again, yielding results that are essentially irreproducible.
So the maximum petaflops of the HPC become irrelevant as they were all expended "quickly" producing a result that is useless.
The call by government to make the necessary Engineering happen is bound to be unproductive if not counter-productive. Commercial users of HPC must demand what they need from the manufacturers; and remain severely unhappy if they don't get it delivered. That is the only leverage that the free market can pull. If users need to form groups to produce common demands, then that wouldn't be the first time in history.
Let the boffins/buffoons at the manufacturers work out how to squeeze performance out of their own hardware because they are the ones who have intimate knowledge of its foibles. Make them provide a uniform application software environment that is portable and produces consistent results in application software.