Docker and storage – solving the problem of data persistence

Containers

For me containers are for serving code, not for data. If you end up running database or document/file/object store instances in containers then you're doing it wrong. I still believe it's easier to have your data in VMs instead of containers.

We have several applications that are pretty scalable running on ephemeral AWS nodes, created when that was the only option, and it's so much simpler operationally. Data backends are not as elastic as frontends, true, though we try to use object/file stores when possible so the scaling is not our problem. Patching and application upgrades have always been a case of just blowing away the VMs and deploying new, so you phase rollout. You also avoid all that legacy and cruft that people dump into directories and never clear up.

I don't see the advantage of running 5x database containers in a VM vs 5x database VMs. Though if someone can explain that to me then happy to reconsider.

Scalability is key

I think the persistence problem has already been solved.

Containers mean it's simple to scale an application be it the front end or the back end and the database element is no different. If you are using a clustered DB its easy to kill of a node or two and still have a fully replicated version of the db on the other nodes. Whole server instances can be ephemeral as long as your DB cluster is large enough.

The bit that does not fit the container paradigm are standalone dbs as by their very nature cannot be easily destroyed meaning they do become pets again. Examples to use rather than traditional state storage techs such as mysql are mongo, Cassandra, rabbitmq, hadoop, hdfs or kafka if your still needing rdbms try crate.

I'm not saying this works for every use case but I'm yet to find one that it could not help with or make simpler to deploy (even if it means more setup and config on your build/ci boxes to get there)

Also is anyone thinking about testing here. That's the real reason I started using docker in the first place. Being able to spin up an entire application stack including database network and configuration and run tests against that (where the testes images are the exact ones that would be deployed) on every build was a bit of a pipe dream before docker. (For us anyway)

Obviously it's just a tool and people can use it in a million different ways but for us its been an enabler to more tested code, configuration management and continuous delivery.

the perpetual problem: how do we keep data + meta-data available, and separate from the code

that was an issue in 1960, and it still is. mainframes, mini's, pc's, vm's, containers, serverless code, the cloud:

all presented as the philosopher's stone that was going to solve all your problems.

all to good to be true, and that's exactly what they all are.

all ignoring the same issue: how do we keep the data and meta data available, even when

the engine blows a fuse... or even blows up....

I get it

I get the deploy benefits, but in my experience, deploying a solution is not the solution, it is merely one aspect of solution delivery, a thorny one admittedly, but no customer has ever paid me to for the deployment. I get the scaling of the application, the ability to just run up additional instances on demand and drop them when they are no longer needed, which leads directly to what makes containerization so good for testing, the ability to blow away what was there and start again clean. However, this is exactly what makes containerization a problem for me; If you are blowing away an instance when you no longer need it, what exactly was the point of doing it in the first place? Stateless activities are (IMHO) meaningless unless they take place within a statefull context, and I don't think Docker et. al. have grasped that.

What about EMC {code}'s RexRay?

This is a well written piece about about the issues around persistent applications and containers. We often forget that while we'd all like our applications to be stateless, 7 of the top 12 Docker containers on Docker Hub all require some form of storage. However, I was disappointed to see that our project called RexRay was omitted from mention from the article. Its been around for over a year now, and aims to provide a solution for running containerized applications that require storage. Please take a look at github.com/emccode/rexray