Are SPC Benchmarks useful?
A commentard ripped into me over the HP SPC-1 benchmark win story - No Dell, no EMC? Well, HP's storage champ then.
Here's what the comment said (between arrow lines):-
You're kidding me right?
Chris, have you even read the specs of the arrays you're drawing comparisons against? Do you understand how the SPC benchmarks work and the impact particular type of resources have on the different workload profiles used?
Storage performance and scalability is large dependent on a number of different resource types and the ability to distribute different types of workloads across available resources (utilization rate).
From a workload distribution perspectives in simplistic terms we can think about frontend and backend. For the frontend, the 3PAR 7400 is using more FC ports than both the V7000 and the HUS150. We could get extremely technical on the impact more ports yields from a buffering and queuing perspective, but I think its pretty clear that more ports ultimately means that a host can process more I/Os in parallel.
From a backend workload perspective it more about the disk to offload the workload to. In some workloads profiles its almost always about the disks, with others parts of the I/O chain in-between almost running at line speed. The 7400 again has more disks (SDDs) than both the V7000 and the HUS150, in both cases we are talking double digits more and at 2500IOPS a pop thats 25,000 raw IOPS we can handle without cache! .
When it comes to dealing with the different workload profiles cache (for all but random read workloads) is king, a well designed array's performance scaling profile is based largely on this single resource type all other things being equal. The 7400 has more than 4 times the amount of raw cache than the V7000 and double the cache of the HUS150 When it comes random reads SDD is our saviour and as I said above, we have more SSD spindles in the 7400.
SPC comparisons that are designed in this way prove nothing. Please stop harping that X is better than Y because you've looked at some summaries at the SPC website, its false (performance) economics and just wrong.
So the SPC-1 benchark result summaries are not a valid way of comparing different vendors' systems.
The SPC web site home page includes this text; "SPC-1 benchmark results provide a source of comparative storage performance information that is objective, relevant, and verifiable. That information will provide value throughout the storage product lifecycle, which includes development of product requirements, product implementation, performance tuning, capacity planning, market positioning, and purchase evaluations. The SPC-1 Benchmark is designed to be vendor/platform independent and are applicable across a broad range of storage configuration and topologies."
The vendors agreed the SPC-1 benchmark and submit systems to it and publish results. So it's valid, I strongly submit, for us hacks to write about them. In other words I disagree with the comment above.
Is that a reasonable line to take?