Infiniband, anyone?
Funny how Infiniband has been offering a switched fabric network with separated control and data planes for about a decade, at a price per port that for a long time was way lower than comparable Ethernet (once the Ethernet specs were even drawn up for the link speeds that IB was supporting). Plenty of supercomputers have had single IB connections from compute nodes to a converged data/storage fabric.
Not sure how much 40Gb Ethernet switches are going for, but considering that a basic unmanaged 8 port QDR (== 40Gb/sec signalling, 32Gb/sec data) switch can be had in the USA for less than $250/port, and a 36 port top-of-rack QDR switch with redundant power for about $140/port, I'd be surprised if there were such low entry points for Ethernet switches with comparable bandwidths and software defined networking capability. Even that tiny 8-port QDR switch can be connected into a mesh fabric, and toroidal IB networks with peer-to-peer links to adjacent and nearby racks can allow some degree of horizontal per-rack scaling for deployments growing from small beginnings that cannot justify more expensive core switching.
Granted, this last point is making a virtue of necessity, in that you pretty much *need* to run an Infiniband subnet manager on an external host once you get to a decent size network. The subnet manager that was supplied on an embedded management host with our modular Voltaire DDR IB switch was not much use, as it tended to lock up... it's easier to restart the subnet manager, or switch to a failover backup, if it's running on hosts you fully control. =:^/