Re: But ...
"If you turn of the checksumming there is no point of using ZFS. The hype with ZFS, the main point of using ZFS, is because it protects your data via checksums. If you dont do that, then use another filesystem together with hw-raid instead."
So you think checksumming is the only reason to use ZFS? And you think hardware RAID is a good solution?
Not only can I read the documentation properly, I've been using it on production Linux systems for the last couple of years, so whereas in an ideal world adding loads of RAM would be great, SSD caching doesn't (on a live system) make all that much difference - and I actually value my RAM.
One of the issues I didn't raise in my earlier comment is that ZFS on Linux does not use the system page cache but instead uses it's own pre-allocated memory. Two problems here, if you use the default settings then after a while it *WILL* deadlock and crash your system - badly - this is a known issue. Second, if you try to set the allocation "too high" the same thing will happen - problem being that there is no hard and fast definition of "too high". (and of course it's a chunk of your memory now available for other apps!)
I really don't care what an investment bank have to say on the matter, if you try to use checksumming there simply is no comparison, without checksumming and if you stop ZFS from using it's dedicated memory for caching, if you compare ZFS mirror to EXT4, you're not going to see better speed .. the better speed comes from ZFS's heavy use of RAM for structure caching.
Just to ice the cake, and the reason I'm browsing at the moment and trying to slow my heart rate, earlier I rebooted my workstation to find my entire system had reverted to a copy from ~ two weeks earlier. I have absolutely no idea what happened, but absolutely everything had revered, and all my snapshots from the last two weeks had vanished, leaving only snapshots from prior days that should no longer exists.
After much fiddling I noticed zpool was reporting one of the two disks on the mirror as being offline .. so I rebooted the system in the hope that it would come back and magically recover things. On booting up it did magically recover things, all my files are back, the old snapshots are gone and all my recent snapshots have reappeared .. yet zpool is still showing the same disk as offline ...
So .. I've just backed everying up (again) and ordered 2 x new 2Tb drives .. this is one of my last ZFS based machines and it's going back to LVM + SW RAID10 as soon as the disks arrive !!!