To both comments - to the first guy saying people moving to local storage - that makes a lot of sense if you can accurately predict your workload and space requirements up front. Or if you have advanced software running locally that can manage that for you (such as Oracle ASM). MySQL by contrast is absolutely terrible with storage management.
Local storage is of course quite primitive when it comes to distributing data, especially as you grow.. Of course some SAN technologies are crap at managing that as well so the idea is to not use a crap technology:)
Of all the companies I have worked for (all of them startups) - none have been able to accurately predict workloads or requirements. So having a dynamic storage system on a SAN really helped out (vs local storage). A simple software feature change could have a dramatic impact on I/O requirements virtually overnight.
I found it funny that these ScaleIO guys say their software is being tested by, among other things "in memory databases" -- where's the disk I/O need there?
To the second poster talking about upgrades - totally agree there too. The Scale IO "Vision" sounds very nice, but it will take a LOT of deployments and years in production before they can convince me it can work.
Many storage companies have tried to go hardware agnostic over the years, and more often then not they end up abandoning that approach - too many things to support whether it is OS versions, drivers, firmware (even firmware on the underlying disks or SSDs). So many combinations increase the risk for such an endeavour to unacceptable levels for anything beyond the most trivial deployments.
So say they come to their senses and standardize on say a small number of proven configurations - the complexity (and latency) involved with such a grid storage system still has a lot more risk for performance problems and availability problems than a traditional tier 1 storage system for online applications. Now if your talking about tier 2 or tier 3 storage, then of course the requirements aren't as strict.
I think Amazon does something similar with their EBS storage - a grid of servers with iSCSI on them. And it sucks, horribly. S3 is a different beast since it is an object store, your not going to be running a database or other transactional application off of S3.
Don't get me wrong I think the idea of ScaleIO is great, assuming they can get it to work right (for that mainly they need years of time). I don't have confidence they can pull it off though, it's a really complex task to get right.