Re: Why do we actually still need a SAN?
"most storage gurus still hate sharing networks with the IP traffic" - Well, they will have to sooner or later, with the increased deployment of converged HBA/Ethernet cards and 10GEth networks.
"putting in extra networking just for the FC-IP traffic" - you don't need that, as the virtualized servers are using DAS storage. Thus, storage traffic is left within the server. Unless you want to replicate block devices or use Gluster etc., for which with you can use EtherChannel (cheap) or a 10GEth network (expensive).
"rapidly run out of bandwidth" - Matt, the highest network spikes are usually on the proxy tier, the largest data movements are usually on the database tier. You put them on different networks anyway to satisfy IT security, and you don't do a weekly offline database backup at 10am on Tuesday morning, rather than the incremental online backup after 11pm every day. If you don't want to do that, it is better to leave the storage data stream within DAS. Strangely enough, you haven't mentioned Dataguard or SRDF over IP, which put load on the network already.
"".....SAN is never fast...." Sorry, but yes it is. " - Matt, compare the IOPS and throughput of a direct attached SSD or SSD PCIs with what you get with a SAN. A PCI bus is simply is faster than the SAN fabric.
"Every direct-attach disk introduce a single point of failure for your data, and also massive inefficiencies in storage utilisation." - Firstly, storage inefficiencies never went away because of the SAN, they were transferred to the storage arrays, as I pointed out in my original post. Thin provisioning is done on OS Virtualized servers, too, such as RHEV. Spindles still pop may the disk array be in the server or in the SAN. A storage strategy removes inefficiencies. And RAID 0,1,5,6,50,60, etc can also be done on the server itself, if you have a sufficient number of drives. Have a look at the Dell c6220 or the many 4U servers out there. In addition, DRBD can help in HA implementations.
"You don't know how to design a SAN" - Matt, one storage controller serving, let's say, 100 servers, is a SPOF, even if you have two Cisco Nexus switches. The way around this is....another storage controller and array, preferably in another data centre. Let's compare the expense of two 4U x86 servers with two IBM XIVs, 4 Brocade switches, etc.
Someone said "You never managed a few hundred servers, did you?" - Yes, I did. And I've seen a whole datacentre going down and weeks of restores/roll forwards because the SAN went pop. Strange, when it comes to security everyone is saying compartmentalization. Not so when it comes to resilience. Have a look at vSphere, RHEV, Canonical Landscape or Oracle Enterprise Manager for managing distributed systems.