Re: ZFS scary bits
Fail over - just don't go there.
It has always puzzled me that Digital (VAX/VMS) solved this so long ago with the VAXCluster and yet it seems to have been a festering sore for everyone else ever since.
VMSClustering was (and still is) actually far more advanced than mere failover. And the source code was (briefly) there on microfiche for anyone to learn the finer points of the technology, just before the lawyers and corporate types moved in and Digital entered its long slide into oblivion. Copyright, not open source, but even so ....
Sigh.