There isn’t anything inherently evil about industry standard benchmarks, just as there isn’t anything inherently evil about guns. You know the saying: “Guns don’t kill people – people kill people.” (What about bullets? No, they’re not inherently evil either.) But in the hands of motivated vendors, benchmarks are weapons to be …
Waste of time
Sorry, but its a waste of time and money.
Most companies don't or wont spend time performing these benchmarks because they won't sell systems and there's too much risk in doing it and not coming out ahead.
Then there are ways to cheat the system.
Switching Centos for RHat. Free vs licensed copy per node for 3 years.
Going with stock Apache Hadoop vs Cloudera vs HW vs ...
How much memory, networking etc ... all impact the performance numbers. Not also is important is that in the marketplace, speed isn't as important as you would think. You can always expand out adding nodes at a minimal cost giving you more storage space and computing power.
TPC-C became useless because you can either go for the lowest cost per transaction, or the highest number of transactions but not really both and once you get sub $1.00 (USD) the incremental improvements are lost and are really due to Moore's law.
Also add to the equation... you have Intel, AMD and maybe IBM's P7 competing.
It also gets worse when you have companies trying to game the system. We once did a benchmark where we specified the hardware. Each vendor tried to tweak the system by supplying more, but smaller disks as a way to get better I/O performance. We told them they had 48 hours to fix it or they would be excluded from the bid.
The point is that these 'benchmarks' are really a waste of time and have little or no value to the decision makers.
- Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
- FOUR DAYS: That's how long it took to crack Galaxy S5 fingerscanner
- Did a date calculation bug just cost hard-up Co-op Bank £110m?
- Feast your PUNY eyes on highest resolution phone display EVER
- Wall St's DROOLING as Twitter GULPS DOWN analytics firm Gnip