Hi! Daniel from NetApp here.
Actually the (big/sharp) hockey stick do say a few things, mainly that the system will not gracefully come “out of performance”, it will dump it on you. But that discussion is for another day.
IOPS is cool but latency is what’s important. The goal has always been lowest/low latency. Or as much IOPS as possible at X amount of latency if you like. It was never a max IOPS race anyhow.
HDS look very good here and they should get some cred and I wont dig into it, maybe someone else will do the detailed work.
In regards to controlling the 100% load results you can. Sort of. But it’s a bit more complicated than that. Easiest way of saying it is that you, as the testing vendor, sets a limit. A latency limit.
For HDD that limit is usually 20ms (30ms is the SPC limit as noted) as that is what most transaction applications consider as max latency before getting angry hence you want to have some head room.
For SSD that limit should be 1ms as SSD and Flash was introduced to drop latency far, far below what HDD can deliver and 1ms is also what all vendors of SSD/Flash systems has as a starting point.
So setting 1ms as the latency ceiling you get X number of IOPS out of the system being tested. The system might be able to deliver 10x the IOPS but not at 1ms or less.
Its not avoiding or cheating the hockey stick IMHO, its showing what a system can do up to a certain latency point. Simple as that.
I know this is a simplified view and explanation and this could turn into a long discussion which I sadly don’t have time for (I have flash to sell. ;) ) so I end it here.
As for other type of tests, the only other I know that has a real life connection is the “VMware mixed workload test” that ESG did. I have not seen them publish that for a while though.
I thought that was a pretty good test; VMware platform and then simulating Oracle, Exchange, Webserver and Backup/table scans/indexing etc at the same time to show how a system would cope with multiple workloads all while being (well) below 20ms for all apps.
Cheers,
Daniel