The performance summit of K2, Kaminario's mountainously named flash array, has been lifted to 2 million IOPS. We have the Fibre Channel-connected flash K2 listed as running at up to 600,000 IOPS and exhibiting an 8GB/sec throughput. Its capacity ranges from 3TB to 100TB. Now Kaminario has tweaked the beast and got 2 million …
You want to play these games? Fine, lets play.
Kaminario's claims described in this article are so ludicrous and in some cases blatantly false it is hard to know where to begin, lets start with the simplest.
The word "system", Kaminario claims their product is a "scale out" not "scale up", in which case when you offer a small product, that, is a single system, and when you offer a full rack product, that, is many systems, plural, if it is one system, then you are scale up not scale out. So Kaminario's results are not for a single system all the numbers they boast about are for a multi-system configuration. We will get into this and see that all their claims are backward, they trail rather than lead in all the configurations they have chosen to discuss.
Then there is the issue of latency..
> The gloating Kaminario also said the system's latency was less than 1 millisecond,
> claiming this was "four times better than the closest competitor".
They don't name the supposed "closest competitor", but I will volunteer myself in that role given that they make plenty of comparisons to our product later on. Now they don't say how much lower than 1ms their latency is, but if we look at the SPC-1 results we see the K-2 at 95% load with 3.7ms read latency, the only results less than 1ms are for the 10% Load Level Test Run where their latency is 420us. A Violin array under similar load percentages would have latencies of around ~800us and ~100us respectively and wouldn't have any of the >30ms latency spikes the K-2 displays. So It appears Kaminario meant to say they were "four times worse than their leading competitor".
It is worth noting that the K-2 SPC-1 benchmark configuration was an entire rack at $490,000 with a whopping 1.159TB of storage, that's a "." not a "," so that comes out to ~$300/GB by my calculation. I'm not going to discuss exactly how many times worse than their leading competitor this is, lets just say its a lot closer to 40X than 4X.
>"This benchmark reinforces the importance of a … scale-out architecture compared to the
> proprietary scale-up systems of TMS and Violin. [It] allows our customers to buy one K2
> and grow it as their needs grow. The equivalent benchmark by Violin required two SLC
> (faster single level cell flash) systems; we scaled to 2M IOPS and 20 GB/sec throughput
> with one MLC system," the firm responded.
Again, if you are a scale-out architecture then your rack is a collection of systems, its hard to say how many systems one should view the K-2 rack as being, 3?, 15?, 33?, 45? but a single system it is not. But that aside lets consider a rack's worth of 12 V6616's, and remembering that their "benchmark" is pure random read, so the rack of SLC 6616's gives 15M IOPs, 60GB/s throughput from 120TB of capacity. Or 7.5X the IOPs, 3X the throughput and 2X the capacity. If we consider a rack of 12 MLC 6632's we would see 12M IOPs, 48GB/s of throughput and 240TB of capacity or 6X the IOPs, 2.5X the throughput and 4X the capacity.
So all this benchmark shows is that in all possible ways the performance and capacity of the K-2 lags far behind the products of their leading competitor.
> If you are going to Oracle OpenWorld over the next few days (it starts today and runs
> until 4 October), you can see the K2 in action, presumably spitting through complex
> Oracle database applications as if they were trivial spreadsheets.
If you are going to OOW over the next few days certainly stop in and see a collection of K-2 systems in action giving 2M IOPs from a 40U rack, which is only 7X worse than their leading competitor, then walk 10 feet across the isle to said competitor and see 2 V6616s giving 2M IOPs from 6U. And read about how that configuration produced the most recent World Record TPC-C benchmark on Oracle, which will will assume is like spitting through complex Oracle database applications as if they were trivial middle school algebra homework.
Founder & CTO
Violin Memory Inc.
Performance is serious business, not a game
The comments written by Violin’s CTO mix apples and oranges from multiple benchmark data points resulting in false conclusions. Let me clearly and factually present the truth.
First, Kaminario did two different benchmarks: an audited SPC-1 benchmark designed to give customers an apples to apples comparison of vendor performance with an industry standard workload, and a second benchmark based on an IOmeter-based random read only workload identical to what Violin promoted at VMworld.
SPC-1, as the market knows, is a well-defined benchmark and has a very high write component (2/3). The result from that benchmark was 1.2M SPC-1 IOPS at a cost of $0.40 per SPC-1 IOP. World records! Clear, factual, audited. Violin is welcome to join us in the peer review of SPC-1 and do the benchmark.
The second benchmark was a random read only workload identical to Violin’s 1.35M IOPS announcement. The difference is we used a single all MLC K2 system and they used two 6600 SLC based systems (as published in their report they measured 750K IOPS on a single system). Our result was 2M random read IOPS to their 750K. Our latency was 0.9 milliseconds (on the host) vs. 3.44 milliseconds in Violin’s benchmark report. Also in the report, they mention 4GBs throughput out of a single SLC 6600 (or 2GBs on their MLC systems) vs. 20GBs in our all MLC system. I am not sure why all of this confuses them. It’s again, clear, concise, and factual. Cheaper Flash, far better performance, single system.
Let me also address Violin’s misconception of our linear scale-out architecture. The K2 is a single system no matter how much capacity you put in it. Start small and grow. It scales out! It’s still a single system, because:
-- Volumes always span the entire address space, across all k-nodes
-- When adding capacity (non-disruptively) the system redistributes the volumes across the entire system automatically
-- The K2 has a single management application and full access to all the address space from any access point
-- It does its own automatic load balancing and does not rely on the application to do that for it
-- It has less overhead since you don’t save hot spares for each system (much better TCO)
Violin wants you to count the number of components in our system. A car has 4 doors, 4 wheels, 5 seats, pipes and cables, a few pistons, a radio and that cute little Hawaiian girl on a spring dancing on the dashboard – it’s still just one car.
Finally, performance is important but the real issue for all flash storage is enterprise capabilities. K2 provides end-to-end, self-healing high availability and fast, high-volume data protection, including N+1 redundancy, automatic failover, high-volume snapshots and hot-swappable components for fast, easy and non-disruptive operations all designed from the ground up for SSDs. With Violin, can customers trust 3rd party snapshots that were not designed for flash? Can customers trust Violin in replacing a VIMM while their live system is running a mission critical application?
We invite customers to seriously measure the performance and capabilities of each of us. Then they can decide who is playing games and who is serious about enterprise storage.
Shachar Fienblit, Kaminario VP of Engineering
Re: Performance is serious business, not a game
mostly factual i guess.
there is merit in both statements. but a bit more detail is needed.
The violin reference is based on 1 or 2 3RU units.
Based on all available information the K2 numbers are based on a full 42RU rack worth of gear what appear to be dell based blades.
However you want compare it, it is not apples to apples. As I read the violin response it would seem that if you scaled their platform to a similar configuration as the K2. their performance is much better..
in a similar context of benchmarking, it is probably a matter of days before a Server Vendor sponsors a TPC-E, TPC-C, TPC-H, TPC-DS or an Oracle or SAP application Benchmark result with Kaminario. As a key element of the TPC Audit Process, primary storage has to suffer and survive a variety of failures with little or no degradation in performance while under peak load.
Now there are a lot of entries on the TPC, Oracle Benchmark and SAP benchmark sites. so I may have missed it but I dont see Kaminario listed anywhere on those sites. While SPC-1 does measure storage performance. It provides no relevance to actually running Oracle, Sybase etc.
But then again - when you are building your kit with components from dell and fusion maybe it is hard to deliver a competitive price point for actual user-space benchmarks....
Perhaps its the challenges with your supply chain the limits your ability to compete in that area. Oracle over comes this by doubling down on the number of arrays (F5100) used to to allow for hardware failure testing during benchmark testing.
Perhaps two-racks of K2 gear is not that cost competitive.
Or maybe its the fact the Dell Blade has to be removed to service the PCIe Fusion Cards within....
Or maybe its the fact the hardware you depend on is limited to your suppliers (Dell/Fusion) and your supply chain and support model is limited by their production schedule and delivery model.
140TB @ 14M IOPs, 140GB/sec - been there -seen that! This was done by TMS long time ago.....
Is this why the Kaminario system was named after the SECOND highest peak (K2)?
Performance is a serious business but some people don't treat it as such.
Kaminario's Shachar Fienblit suggests that this is not a game and that I am mixing apples and oranges from multiple data points to produce a false conclusion, I have to disagree.
Kaminario is not just mixing different benchmarks they are mixing different products. Their SPC-1 result is not from the MLC K2 product (previously called the K2-F), but from the K2-D, an all DRAM product that is no longer listed as a product on their web site.
Both in the article, in their press releases and in their response in this thread Kaminario switches back and forth quoting both benchmark results and saying they come from the "K2" without any hint that there are two totally different products involved. Even the link on the SPC-1 results page which is labeled "Kaminario K2-D (1875K-1.1)" leads instead to the K2 Flash SSD product page.
It is hard to see how such phrasing as they use coupled with dropping the -D and -F suffixes from the product names and linking to a different product than was submitted for the SPC-1 benchmark could do anything but lead people to draw false technical conclusions of the benchmark performance of their product(s).
Hopefully I have made my point about Kaminario's benchmark claims sufficiently clear at this point which was the intent of my original response and so I will not dwell on the rest of Fielblit's response which strays from the technical into the realm of marketing.
However with regard to Fienblit's last statement, which clearly falls into the realm of the technical.
"Can customers trust Violin in replacing a VIMM while their live system is running a mission critical application?"
The answer is an emphatic 'yes you can!', as you can see demonstrated here.
Founder & CTO