Various PCIe solutions are under development
There are a number of options being developed for extending PCIe outside of the server and switching it, using low-cost 'extender' cards with both copper and lightweight, flexible optical interconnect options. Most of these solutions provide a substantial reduction in cost and power over populating NICs and HBAs directly in the servers, while providing the latency and bandwidth benefits of staying in the PCIe domain.
Traditionally I/O has been way underutilized. Server virtualization is turning that around and leading to increasing I/O needs (more or higher bandwidth NICs and HBAs). But perhaps more important is that I/O is often very bursty, and varies with shifting loads and applications. This means that some I/O can be swamped while other I/O is horribly underutilized. The best way to overcome this is to virtualize I/O, not just across VMs in a single host, but across many hosts, and dynamically assign, monitor, manage and reassign that I/O as required. Using inexpensive, high bandwidth, low latency PCIe connections in every server as the 'in-rack' fabric, interfacing to whatever data center network you prefer (including CEE/DCE and FCoE) is becoming a viable alternative.