Re: purchase price is not the full cost
I haven't really looked for success stories with Oracle's - but I see they have showcase video with Specialized (the bike company). So it must be good, right? ;-)
I do zfs sends from one machine to another, hourly. In case of a failure, I'd need to switch the NFS mounts manually.
For our customer, that's good enough. Also, because the delta is usually small (though the data-set is 3+ TB)
I use FreeBSD and I was thinking about HAST - but I'm not sure it's really a solution or if it doesn't create more problems than it seems to solve.
Every two-headed HA-"Solution" lacks the "arbitrator" that can decide if a failure has actually occurred.
GlusterFS will officially support FreeBSD, soon - but it remains to be seen how well it actually works. We do use it on Linux, but we don't really have a tough use-case, yet.
From what I have read, lots of small files just kill GlusterFS and thus the traditional "home-directory" use-case is still only really served by NFS - which is why all kinds of HA-solutions come into play in the first place.
In the cloud world, high-availability is moving into the application-stack anyway (like with RIAK, where you have stuff distributed between multiple servers anyway, so you don't need to cluster the individual servers).
But for now, there are way too many "legacy" applications out there that require a file-system - and assume it's local and "always there".