Getting higher efficiency on Ethernet
It's important to recognize the difference between typical Top500 entries and what Amazon has accomplished here. This EC2 offering is a real live "HPC in the Cloud" implementation with layers of abstraction that you won't find in classical HPC/Top500 environments. And even with those additional capabilties, the efficiency demonstrated by EC2 is very competitive with efficiencies you will find from many of the Ethernet clusters on the Top500.
But to infer that Ethernet can't deliver high efficiency for HPC is incorrect. Take a look just a little bit lower on the Top500 list (http://www.top500.org/list/2010/06/300) and you will see at #208 there is another 10Gb Ethernet cluster running iWARP and it is delivering 84% efficiency - so, only 16% of the FLOPs "go up the chimney" - which is even better than many of the InifiniBand clusters on that same list. The use of iWARP (http://www.intel.com/technology/comms/iwarp/index.htm) is what helps deliver the high efficiency.