Q2 wins drag Cray back into profit territory

It is indeed the interconnect that matters. The interconnect on Fujitsu's K computer is superb, and is wholly responsible for that machine achieving its very high sustained performance.

The problem is that a some important compute loads are not infinitely divisible. And even for those problems that are highly parallel there is a law of diminishing returns. Using smaller and smaller CPUs means that the interconnect has to do that much more to compensate; you've got to get data in and out somehow. Ultimately the computer becomes all interconnect and comparatively little compute. Losing access to cheap high performance CPUs would make the interconnect problem for Cray, etc. a whole lot harder to engineer.

AFAIK the K machine achieved a pretty good balance between interconnect and single thread performance. It's mean/peak performance ratio is pretty close to 1, something you can achieve only if there is adequate interconnect performance for the CPUs used. Fujitsu built the interconnect right into the CPUs, which is expensive but a good way to go. I don't know what Cray do. Looking at the mean/peak ratio for some of the higher up entries in the Top500 list, you'd have to conclude that they have sub-optimal interconnects.

Interconnects themselves are not cheap to develop, and it will all become Ethernet one day anyway. The SerialRapidIO standard can't go anywhere because the market for it is too small to fund the development. Same for Fibre Channel. Speaking purely hypothetically, how long before PCIe becomes too expensive to make it compete with Ethernet? Several lanes of 400Gb/s Ethernet sounds a lot faster than several lanes of PCIe... A 400Gb/s Ethernet switch chip is going to cost a huge sum to bring to market, and so would a comparable PCIe crossbar; can the world afford to do both?