Oracle/Sun Solaris ZFS is the Gold Standard

Chris Mellor asks, "Which storage OEMs will embed Ocarina or Permabit deduplication code in their products?"

The answer is... who knows!

De-Dup and Compression source code is free, available on the internet, and no one in their right mind (unless on a religious crusade or on a severe catch-up plan on their own proprietary code) would develop a new system to leverage for an embedded storage product when a half-decade of open source development has already achieved so much.

Oracle/Sun Solaris ZFS is the Gold Standard now for embedded systems.

Data reduction is not a matter of ethos, it is a matter of performance

As far as VM storage is concerned de-duping it leads to drastically reducing the effective IOPS load. The "common part of the VM" + "differences" can actually fit into tier-1 storage or even cache. As a result the performance is much greater compared to a non-deduped install.

Microsoft?

So, what is the Great Satan of Computing doing about de-dupe? It would knock a big hole in NetApp's Windows-centric market if M$ came out with a proper de-dupe capability in the next gen of Windows Server. Who'd bother with clever second-gen de-dupe if M$ just bundles it in with the OS? After all, the majority of today's de-dupe, especially in backups, is in Windows servers and desktop files. That would leave the rest fighting over the UNIX/Linux and mainframe market.

RE: Maybe I've missed something - ZFS can protect against it

Super Fast Jellyfish asks, "I copy from disc C to D (even if they are virtual) to protect from bad sectors, is the de-dupe going to point to the original data and lose the effectiveness of a disc backup?"

If your desire is to protect against bit-rot, copying data between volumes is no longer needed under ZFS, there is a "copies" clause to perform this action automatically.

If you want to protect from bad sectors with ZFS when running dedup on a single drive, you can run with "copies" property, and always make multiple copies of your disk blocks. With ZFS checksumming your data, it will detect the bad block and fix it dynamically upon discovery (or upon the next scheduled scrub), instead of subjecting the user and/or business to "bit rot" data loss, as other operating systems do.

If you have multiple disks and running with redundant physical disk setup, the copies property is not required to protect/correct from bit rot, but rather the spare disk[s] will manage it under ZFS.

The next item, you can apply compression, to mitigate the expanded disk space required with "copies" property or redundancy at the physical disk layer. This also reduces performance by reducing the quantity of data being read off the disk when there is ample CPU capacity.

What is really nice, dedup sits on top of this entire infrastructure, to increase performance, and reduce disk usage. On virtual servers, this will enable more secure data, faster performance, and massive disk space reduction requirements.

If there is a problem with user error, with unlimited rolling snapshots, you can even recover without a backup if the physical media has not been compromised..

If there is an outstanding legal question with NetApp with ZFS, it will also apply to non-production quality Linux BTRFS as well. Oracle, the sponsor of mature ZFS & immature BTRFS, is big enough to purchase NetApp, anyway.

ZFS is the Gold Standard in storage and reliability.

I am uncertain how Ocarina & Permabit deal with bit-rot on the disks or whether de-dup on non-ZFS infrastructures with (or without) multiple disks can be protected against.