Wow, still using disk and PCIe storage? You look like a flash-on victim, darling – it isn't 2014

> "The disk-based IO stack in an OS takes too much time and is redundant"

Yeah, who needs all that rubbish like metadata, consistency and replication?

The problem with persistent memory at the moment is that it requires code-level changes (see http://pmem.io/2016/06/02/cpp-ctree-conversion.html for example) if you want to have this great performance.

A thin filesystem-like layer would lose you a bit of speed compared to writing to bare memory but give you an awful lot more back in terms of reliability and management. Persistent memory-native apps will come, but not for a while yet.

resilience

This is going to push the need for resilience on DIMMS, and that's likely to be erasure coding. Unfortunately that'll then stomp on the CPU and we'll need another upgrade cycle :)

This is like Violin when they emerged - a few niche environments will need it, the rest of us will salivate but ultimately make do with the normal technology which is still fine for running the mail server, just like the 486 box in the corner was...

Don't forget the processing effort

Don't forget in these calculations that the CPU doesn't spend 100% of its time reading and writing data from external storage. In fact, with a decent amount of DRAM and clever data mapping, the processor might not read/write that often, depending on the application.

Also, we have to bear in mind that when the processor does 'do work", it may be at the request of an external call (e.g. a website) or some other user interaction that takes time over the network.

All this means the delay from storage I/O might not be that impactful, if we have enough other work to do. Hurrah for large memory caches, multi-tasking and parallel programming!!

This is quite a simplification. There are plenty of situations where jobs are using remote storage (SAN say) to access large datasets, which can't possibly be replicated locally due to the data size. In these situations the local disk is hardly hit, and file access is normally sequential, so the criteria you are quoting (access speed, or basically random access latency) is totally irrelevant.

I'm sure there are applications that this sort of architecture would help, but there is plenty of stuff where this is not the case.

tiny niche

The stuff that needs this kind of tech is a tiny niche. Been supporting ops/dev type shit for 15 years now and at really no time has something faster than SSD been even thought as a "nice to have". Current org is a typical online e-commerce business doing well over $200M/year in revenue running standard web stacks (with mysql as the DB of choice). In excess of 90% of all I/O for OLTP occurs within mysql's buffer cache. Disk I/O to the LUNs mysql lives on is tiny, really only happens for very infrequent big expensive stupid queries but far more often for log writing (binary logs, relay logs etc). Storage array I/O workload is over 90% write under 10% read. We probably peak at 15-20% of the array's I/O capacity (meaning it will never be a bottleneck in lifetime we will be using it for).

I was at another company several years ago who, many years before that wrote a proprietary in memory database for behavioral ad targeting. We had dozens of servers with massive amounts of memory(for the time) and clunky processes for loading data into these in memory instances. While I was there they eventually decided that in memory was too expensive so they re-wrote the stack so it did some layer of caching in memory(reducing memory usage probably by 80-90%) and the rest of the data sat on a NAS platform connected to a 3PAR storage array(with SATA disks no less). So they went from in memory to NAS on SATA, and the performance went UP, not down.

I still believed it was a flawed architecture I would of preferred them use local SSD storage (at the time I wanted them to use Fusion IO). But they architected it so it HAD to reside on a shared storage NAS pool it wasn't going to work on local storage. Too bad, missed opportunity.

So I've seen and managed traditional mysql (and Oracle) and vmware etc as well as in memory systems across now thousands of linux systems primarily across many companies over the past 15 years, and solutions like those presented in the article I believe are very very niche(relative to the size of the market as a whole). I could see things changing if the costs get down low enough that it makes no difference in cost/complexity if you are using DIMM SSD or SAS SSD(or even PCIe SSD), but I don't see that happening anytime soon.

I do believe there is a market for this kind of stuff just much smaller one than articles like this seem to portray.

It's Time

Periodically in this industry we get a chance to combine some new hardware and software advanes. The we jump up and down hard on the software stack to remove many of the abstractions to get closer to the (new) metal. Then the layers begin accumulating anew. It has been ever thus.

Nambiar's Law of IO

See http://www.odbms.org/2016/06/nambiars-law-of-io/

Nambiar’s Law of IO

BY ROBERTO ZICARI · JUNE 29, 2016

Nambiar’s Law of IO

BY Raghunath Nambiar.

Two of of the most popular questions asked in the storage performance world are:

– How do you estimate the IO bandwidth requirements?

– How do you estimate the IOPS requirements?

In other words how much IO can be processed by today’s data management and datastore platforms?

Here is my simple rule of thumb, let me call it the “Nambiar’s Law of IO”, #NambiarsLawofIO:

100 Mbytes/sec per processor core for every 7 years of software maturity for IO bandwidth intensive workloads. Examples: enterprise data warehouse systems, decision support systems, distributed file system based data processing systems etc.

10,000 IOPS per core for every 7 years of software maturity for IOPS intensive workloads. Examples: transaction processing, key-value store, transactional NoSQL, etc.

for up to 21 years, beyond which there is no significant improvement, that’s what we have seen historically.

Examples

Platforms in the >21 years category:

Traditional relational database management systems (Oracle database, Microsoft SQLServer, etc):

– Data warehouse: 24 (number of cores in today’s popular 2 socket x86 servers) x 100 (IO bandwidth that can be processed the by software) *3 (>21 years) = 7.2 GBytes/sec

– Transactional: 24 (number of cores in today’s popular 2 socket x86 servers) x 10,000 (IOPS that can be processed by the software) *3 (>21 years) = 720K IOPS (small block random r/w)

Platforms in the < 7 years category:

New generation distributed datastore systems:

– Hadoop: 24 (number of cores in today’s popular 2 socket x86 servers) x 100 (IO bandwidth that can be processed by the the software) *1 (<7years) = 2.4 GBytes/sec.

– Cassandra: 24 (number of cores in today’s popular 2 socket x86 servers) x 10,000 (IOPS that can be processed by the software) *1 (<7years) = 240K IOPS

https://www.linkedin.com/in/raghunambiar