Latency. Low end-to-end latency is just as important as bandwidth. That is a major reason that early compute clusters discarded ethernet. You can provide a channel a mile wide in terms of signalling rate and still not achieve application throughput if the latency is too high.

GigE usually uses TCP/IP - and this protocol, while tremendously redundant & wonderful for convergence, is not terribly efficient at higher capacities.

Infiniband is a beautiful thing for high performance & clustered databases. Clustering memory caches between nodes requires high bandwidth and low latency - otherwise performance & scalability is limited.

I would have been interested to see a comparison between Voltaire and SUN as far as Infiniband switches in terms of capacity, size, and latency on a flattened network... instead of only Cisco.

http://www.sun.com/products/networking/infiniband.jsp