WAFL, fragementation etc.

"NetApp has no intention of adding write caching, as its testing has demonstrated a lot of overhead and no real-world benefit."

I thought that WAFL did, in effect, perform write caching anyway as it assembles random writes into what is essentially a sequential one (ideally a complete stripe write). However, surely it can't keep all the clients waiting until it has assembled such a write.

However, the observation that WAFL turns random writes into sequential ones is, of course, true. The one problem is that the if you have a randomly updated file (such as a database supporting an OLTP application), then that very same WAFL operation will fragment the data. That doesn't matter much if you only generally access data randomly, but if you want to access it sequentially then what eventually happens is that large, sequential reads get turned into lots of random ones. If the backend can't cope due to the number of IOPs then it can spell trouble. With housekeeping, backup and so on you can overcome much of this with NetApp tools, but if it comes down to application access patterns then it can spell trouble. That won't happen as much if your data is updated sequentially, but it's the downside of WAFL.

However, on the main subject, for de-duping system images it's undoubtedly the best solution at the moment and the "boot storm" issue is a particularly important one where caching with de-dup helps, although it won't get you out of holes caused by lack of controller throughput or network capacity.

An EMC-centric view

Sent to me:-

"DD BOOST is there to reduce or eliminate the network link between the backup server and the DD Restorer as a bottle neck. DD systems can see a throughput increase because of that but they're never going to exceed the performance of the CPU in the controller as that does everything. We're I a cynical man I'd say NetApp can't do anything about thin network links reducing throughput into their Filers without buying a Riverbed or SilverPeak box while BOOST on the other hand can.

"Also, dedupe loses out to boring old compression in nearly every use case that isn't backup or VMware.

http://communities.netapp.com/message/24939

"My testing confirms what the NetApp guys say in this thread about getting less than 10 per cent reduction, a customer and I got about a 5 per cent reduction after running ASIS on Exchange volumes for what took days, but managed to get a 40 per cent reduction on the same dataset on different volumes by using compression. So people will have to offer more than one data reduction technology as dedupe isn't a magic bullet and the more full copies you have the more efficient it is."

DD BOOST is there to reduce or eliminate the network link between the backup server and the DD Restorer as a bottle neck. DD systems can see a throughput increase because of that but they're never going to exceed the performance of the CPU in the controller as that does everything. We're I a cynical man I'd say NetApp can't do anything about thin network links reducing throughput into their Filers without buying a Riverbed or SilverPeak box while BOOST on the other hand can.

Also, dedup loses out to boring old compression in nearly every use case that isn't backup or VMware.

http://communities.netapp.com/message/24939

My testing confirms what the NetApp guys say in this thread about getting less than 10 per cent reduction, a customer and I got about a 5 per cent reduction after running ASIS on Exchange volumes for what took days but managed to get a 40 per cent reduction on the same dataset on different volumes by using compression. So people will have to offer more than one data reduction technology as dedup isn't a magic bullet and the more full copies you have the more efficient it is.

... Chris.

EMC Celerra deduplicates active files on primary storage

Disclosure: I'm an EMC employee.

NetApp isn't the only one in the market doing capacity reduction of primary storage.

EMC Celerra Deduplication will perform file level single instancing and compression of multi-terabyte active files (VMDK, VDI, or other wise) and it'll do so at the file level.

Doing it at the file level means you can force specific files to always be space reduced regardless of the reduction schedule for the volume or decided to exclude various file types if you so choose.

IT Depends!

Lets keep in mind the age old answer in IT - IT Depends!

First to say that NTAP is "Killing It" is WAY over played. This is someone not getting the facts and just trying to pump technology - in my book this is called FUD. How do I know this? Because I work for a company where 75+% of our data compression solution sits in front of NTAP. OUR customers are telling us that ASIS gives anywhere between 9% to 15% optimization on primary because there isn't a lot of repetitive data. They are getting 75% in VMware environments with Storwize.

Now I will admit that if you have 100's of VMs instances (with OUT the users data stored in the file) then dedupe is best - why compression 100 systems to 25 when I can dedupe it to 1. Makes sense - BUT if I put the data in the .vmdk - then that blows the theory out of the water. It also begs the question - if I put the data on a file share why can't I do both - dedupe the .vmdk images then compress the data? Two great tastes that taste great together right?

Also, lets keep in mind - I don't care how much resources you throw at it, doing deduplication - or 'traditional' compression (not real-time random access compression) you will never get the performance you need (sorry Zilla) to do compression on primary storage. Now if you can with Storwize - why not compress - get your 75% data optimization and then dedupe after - getting even more benefit?

Remember there is never one right answer in IT. Let's stop the FUD and help users find the most robust solution(s) available.