The vast majority of SANs are not limited by network performance
- DIsclosure NetApp Employee Opinions Are My Own -
1. The network hasn't been a bottleneck to storage for a very long time, unless a moron designed your SAN, or you are really cheap about your SAN switch gear.
2. most SAN's I've checked use about 5% of the available bandwidth. Even for what are considered "enterprise class disk arrays" 100,000 IOPS at 1ms response time and 8K blocks and you're looking at approximately 781 MiB/sec. A single 10GB ethernet connection can handle that with ease. All Flash arrays with "1 meeeelion" IOPS might require around 8 of them.
2. The additional latency incurred by a modern network is currently measured in microseconds, this does make a difference in high frequency trading and some fraud detection systems, it also makes a difference in raw sequential throughput for HPC applications. These are currently edge cases in the "second platform" which is likely to be the vast majority of IT spend for the next decade or so,
3. Extending PCi as a network ... that was what infiniband was designed to do, in some respects, PCi is just a cheap and nasty local only version of infiniband. Personally I've been a fan of RDMA technologies for about 10 years, and RoCEE also looks pretty good.
4. Any of these "Server SAN's require a fast well designed east-west "Server Area Network" .. which was what a "SAN" was before it became a "Storage Area Network" .. ie. you're still accessing data over the network for any "non-local" data.
Having said that there is some merit to using local, ultra-low latency well designed east-west server network, but overcoming the "IO Bound SANs" is not one of them. If you want to pull a server area network to aggregate the CPU and storage class memory resources of multiple disparate computing resources into something that looks like one large physical resource, then its all good, but for the most part, servers, storage and networking all have independent scaling requirements and keeping them separate still makes economic sense provided you have effective ways of dealing with the additional complexity.