Re: Wow
I tested it with a file server, mail server, web server, and (legal) Torrent client on a 1Gbps connection. All together there's about 200 to 300GB of live data for a 400GB L2ARC partition. Three days later, the L2ARC was empty and the spinning disks were handling all activity. Doing diagnostics found that ZFS mistakes even moderately large files for uncachable streaming data. I switched off that 'feature.' Now the L2ARC was populating, but only at a few MB/hour. ZFS's default L2ARC population rate throttled to a peak of 8MB/sec. To be exact, it's 8MB/sec * duty cycle - essentially nothing. That has to be tuned too.
After hours of effort and two days of warm-up, the cache is working well enough that the spinning rust is free to spend time on things that really aren't cachable. I feel like overall the tuning is fragile and shouldn't be trusted.
The spinning rust is a RAID of the oldest disks I have - 10 year old 2TB WD Green drives, I think. Higher capacity new disks go into the backup system. Older disks taken from the backup system go into the server. It's correct from a storage sizing perspective but backwards for performance. That's why I was looking to ZFS for help.
Biting the hand that feeds IT
