Object Storage: A solution in search of a problem?

1. No standard - er, CDMI?

Sorry to say it - this is a pretty scrappy article. Not well thought out.

You leap about between discussing object stores, and then comparing them with the underlying technology - you mention 'tape' several times. The actual mechanism for storing the data is separate from the data.

I was at a talk recently by DDN and was very impressed by their Web Object Store.

Similarly a lightning talk by Hitachi at Cloudcamp on object stores. It is an idea whose time has come.

The problem with "object storage" offerings is that they are all based on slow and bloated subroutine libraries intermediating data access, reducing performance to a crawl. Nobody wants that.

They are also based on opaque proprietary standards that transforms an otherwise-normal data storage system into a data prison you can never leave (or backup). No one needs that either.

Object storage...not so problematic

I suspect the article was written to generate more heat than light on the subject, but storage vendors do have an industry association in SNIA and SNIA is driving the CDMI standard into the storage market at a pretty good pace when it comes to standards. That said, AWS S3 is the de facto standard API for object storage and CDMI will be able to work in conjunction with it.. I spent the better part of last year evaluating a handful of vendors who provide object storage software to enterprise customers and partners to build private and public storage clusters. Each of the vendors is venture backed and their founders all have significant history in dealing with data storage requirements that were not solvable with the traditional file and block storage technology that we've had around for decades. The incumbent storage vendors have taken out their checkbooks and bought the storage technology they think will allow them to participate at-scale in the market. Whether they can be price competitive remains to be seen given their past history of bundling their storage software with their proprietary hardware. Because the object storage market is relatively new, you can expect to see some participants get acquired and some incumbents to change course. The recent Dell announcement about ending their OEM deal with Caringo has more to do with the internal players at Dell than it does with the quality of CAStor. All of this is par for the course and it may be quite a few more years before the object storage market and players coalesce. Remember that there were once over 200 vendors engaged in the manufacture of hard disk drives. After 30+ years we have 4 of them left. The same thing is going to happen in the SSD market too. There is a market for object storage and it is being met by offerings from relative new companies as well as the incumbents. It is not a matter of a technology in search of problem.

lack of demand

I think a lot of it is lack of demand. There is demand of course in large service providers -- those places often have resources to do a lot of things on their own and not rely heavily on enterprise company products. The sheer customer count that needs object storage I believe at this time is quite low (those that feel a need to have it in house). Hence not having much adoption of any of the enterprise object storage technologies.

My company for example uses some S3 for things - but total usage is overall quite small and doesn't justify the need to have object storage in house. Really I think until your into some decent scale with I'd wager at least half a PB of data projected to be stored would folks really get into looking at object storage vs regular NAS which of course is far simpler to manage.

Also don't forget Red Hat storage - from a support and tech standpoint they seem to have a pretty decent and flexible offering. I would not use it as a file server or as a transactional storage system as their marketing sometimes try to convince you to do but as an object system it seems ok.

Just because you can theoretically build a multi-petabyte cluster to store hundreds of nodes' worth of data doesn't necessarily mean you should. Object storage is and probably always will be a niche product, but there are numerous advantages to not putting massive amounts of unstructured data on a filesystem; the first, obviously, being that you don't have to worry about designing and maintaining an artificial structure around it as your data pool grows. You don't have to worry about errant processes on your storage boxes doing things they shouldn't on data being stored as regular files. You don't have to worry about the intricacies of the underlying filesystem, or whether you'll be able to grow your clustered filesystem indefinitely while maintaining scale. You don't have to worry about whether a software bug in a very complex system like Gluster will cause an unrecoverable error or split brain, because object storage tends to be fundamentally simpler than layered filesystems and storage protocols working together to produce the same result.