Mike Stonebraker is famous for slagging Google's backend. And now he's slagging Facebook's too. Last week, in a piece from our friends at GigaOM, Database Grandpoobah Mike Stonebraker announced that Facebook's continued dependance on MySQL was “a fate worse than death,” insisting that the social network's only route to salvation …
Sharding is not that complex
Stonebraker can say whatever he wants - it doesn't make it the truth. You can read the comments in the original GigaOM post to see what some smart people are saying about his claims (which are off, non technical, and just FUD).
(Disclosure - I work for a newSQL company called ScaleBase). Now if you want to shard you data - there are allot of companies that do that transparently, one of them is ScaleBase. It's possible to shard with relational databases, and it definitely let them scale. Moving to a new database is much crazier than sharding an existing one (unlike what Stonebraker said).
Limit of sanity?
While I agree that moving to a new database is crazier in most circumstances, certainly there has to be a pain threshold beyond which the scale of the project would dictate moving away from sharding. It's really just a matter of scale and, for most, sharding allows enough scale to avoid the pain of a DB switch. But 9000 memcached instances? Where's the proverbial line to be drawn?
The uber system
Considering that a lot of this problem stems from trying to take a problem that might have in other days been created as a more distributed system, sharding as a general approach may not be such a bad idea. You are trying to store the entire world in a relational database when degrees of separation limit relationships.
On the other hand, you have Zynga silliness that drives random connections between people across the planet. If the likes of Farmville didn't encourage everyone to have 10x the number of random "game friends" as real friends, the problem might be more manageable.
I've read somewhere that Facebook uses 65,000 servers so has to be true if they have 4000 mysql servers going to 9000. Probably the same data can happilly run in 2-3 proper commercial server databases like SQL Server or Oracle (ok it may require 250 cores but will happilly run). Don't forget the whole point about a database is query optimisation, you don't have that in your engine you are dead. Not all SQL is the same.
This is fun to watch, but irrelevant for almost everyone
The trouble Stonebraker has is that basing a business around very-large-scalability limits you to very-large customers. And there aren't very many of them.
Back in the real world, no small/medium-sized business will want to spend $15,000 a year on a license. They just need straightforward solutions, that work reliably and predictably. Scalability would be nice, but will be a long way down the wish list for almost everyone.
In fact this is symptomatic of the Cloud nonsense as a whole: a focus exclusively on scalability that people don't need, whilst neglecting all the things people do need. At least Stonebraker recognises the need for ACID compliance, but without licensing options for SMEs (and they're commercial, so the freebie community version won't apply), I reckon his business model is doomed.
There is no such thing as the "SQL model". There is the relational model and ACID compliance. And then you optimise for the business case - reading, writing or both. The relational model is pure mathematics so it really can be about finding the best fit to hardware and a good database will use all available memory for optimisation.
Stonebraker senile? Hardly
Having worked with his Illustra crew while at Informix, I'm going to give Stonebraker the benefit of doubt.
Sure it took 5 years for Informix to absorb the extensibility of Illustra and to make it scale. But it was done and the current release 11.7x definitely has a lot going for it. (Except that Mills and company at IBM still baby DB2... ;-) [Yes I'm that Gumby and yes I'm an Informix bigot]
The point is that Stonebraker knows his stuff and he's actually right in some of his comments.
Sharding data isn't a good idea.
YAY Informix, great dB that is. People these days don't even know about cooked and raw. Though this chap certainly does and then some.
Nobody says he does not
The flamewar masks the actual conflict here.
What Stonebraker advocates for is essentially two-tier architectures. Front-end talking to super-scalable back-end which directly manipulates data. No baby-sitting middleware.
What facebook and everyone who wants to scale do is three tiers - front-end, middleware, database. In most three tiers the middleware does LOTS of work in terms of data availability, integrity and performance (it is an ecumenical matter where memcached sits, but IMO it is a part of the middle tier).
In reality Stonebraker is probably right technically.
However, similarly he is definitely wrong in terms of realities of life. 99%+ of the staff you can hire cannot and will not learn how to talk to the ACID backend and _WANT_ the middleware so they can get their work done. Similarly, 99% of software architects and project managers _WANT_ the middleware to ensure that developers do not do something vehemently stupid with the data.
As a result, like it or not the middle tier is there anyway. If it is there however, you might as well make it do a few things which in Stonebarkers abhors.
"People these days don't even know about cooked and raw."
Spotty, fresh-off-the-training-course SAN array admins certainly don't in my experience.
Put it this way, Informix doesn't like it very much when some fresh-faced eejit recalls the course-imparted wisdom of "disks that do not have a filesystem on them are not in use" and moves all its raw storage onto another server somewhere else.....
Paris, the only other other thing alleged to go down that spectacularly.
 Although, to be fair to the poor-bastard-cowering-in-a-hail-of-shit concerned, presumably the trainers employed by a certain very large storage vendor are also unaware of this distinction.
The problem is that there is a large percentage of IT folks who don't think about their work once they start down a certain path. That's the point Stonebraker is right on that point. You get someone started down a path working with MySql, the end up deciding that extending or modifying MySql is easier/cheaper than redoing their infrastructure.
Not all problems are best solved by relational databases. Also some problems require customized solutions in order to scale. What ends up happening is that you have to make compromises in order to create a generalized solution.
Sharding a social graph is exceedingly difficult, because ther is no way to partition the social graph cleanly, and ther will always be croo-shard links. Face book's engineering challenges are far tougher than even Amazon's.
As for Cassandra, which is based on Amazon's Dynamo (and indeed done by some of the same people), even though Facebook originated it, they are no longer using it themselves.
Sharded MySQL may not be fashionable, but it is well understood and mature, which is why Facebook engineering still uses it. There is something to be said about a predictable system, and Facebook has a demonstrated record of operational excellence. This is no coincidence.
Never heard of him!
So much the worse for you. Ignorance is not a point of view. (with thanks to Scott Adams).
ahh Scott Adams...
...I used to love his text adventures
check the price first gents..
..and now back to my MySQL console.
Whether Stonebraker is speaking truth or blowing out of his ass...
I hope there's some minimal level of abstraction layer between the Facebook code space and the underlying database engine, just in case MySQL doesn't cut it (if not now, in five or ten years time). Because it would be a bloody messy job to:
(a) Use grep for every SELECT, INSERT, UPDATE and DELETE statement in the code.
(b) Translate code into equivalent for other database of choice - SQL, NoSQL or NewSQL.
(c) Optimise the result.
Or have the Facebook team been hard-coding their statements willy nilly without even the benefit of using string constants? I hope not. I really hope not. It's one of the canonical "Do Not Dos" in computer science.
He's just pimping VoltDb. MySQL seems to be scaling just fine for FB.
This is all pretty content-free
1) Facebook has lots of users and lots of data
2) Facebook uses MySQL to store data
3) Things become slow (surprise!)
4) Stuff gets moved into faster volatile memory (using memcached, surprise!)
Stonebraker chimes in, saying:
1) The above is bad (WHY??)
2) Dropping ACID properties is bad (this depends on the application and he sure knows that you can drop these properties them _somewhat_ in return for speed until you get distributed hashtables, which may or may not be ok in your application, see also Brewers CAP theorem http://www.julianbrowne.com/article/viewer/brewers-cap-theorem for a discussion)
3) "NewSQL", in particular his VoltDB to the rescue. (HOW?? What does he drop in VoltDB to solve performance problems? What part of ACID is being killed?)
It's like a breakfast cereal addy, FFS.
P.S. I always get into a huff when SQL is brought up anywhere in these discussions. SQL is just the crappy insanely dumb query language on top of whatever your database is.
"I always get into a huff when SQL is brought up"
Sounds like someone can't cope with declarative programming. Or maybe you just don't like standards. I assume you do all the hard work of query processing in your application logic, yeah?
@Destroy All Monsters
> SQL is just the crappy insanely dumb query language on top of whatever your database is
What an utterly idiotic and ignorant statement to make.... SQL is not crap. SQL is not dumb. SQL is the fastest method to crunch data on SQL-based databases. And it can scales very well.
Of course - this also depends how SQL is used. A simple thing like using bind variables is often ignored. SQL used as a mere I/O layer like one would treat kernel device read() and write() calls.
That type of ignorant use of SQL... no wonder your application and database performance sucks.
On my databases - runninng a 1000+ SQLs per second is the norm. Not the exception. And there's no way in hell that you could ever get that performance by pulling db data into a client process, crunching it there and ship it (across process and memory and even h/w boundaries) back to the database.
The biggest monster of all?
that is true ...
"SQL is the fastest method to crunch data on SQL-based databases."
however the implied assumption that SQL is the fastest method to crunch data on databases , or even relational databases, is false. Try kdb+ with its memory tables, functional language, runtime that wholly fits in CPU cache and q-sql on top of it, that's fast.
SQL is the fastest method
Implying SQL is a "method"
> On my databases - runninng a 1000+ SQLs per second is the norm
Maybe you should become a rapper.
Facebook don't do ACID
Only a fool would put data they can't afford to lose on Facebook. Considering that FB are making arbiitrary censorship decisions if someone quotes a song lyric there regardless of fair use, it seems a bit daft to use FB for much more than idle chat with people close to you and which you don't mind that much if what you put there leaks beyond your intended context. I'm sure Stonebraker's comments would be highly relevant to large scale financial applications of the kind I can imagine, but they don't seem that relevant to Facebook.
1. Think about the polar bears! This madman Stonebraker wants to melt the icecaps, but Facebook knows how to slow global warming and still keep all our inappropriate pictures on-line. Good thing, don't know what I'd do without them.
2. SQL is an OK lingua franca, but it is not without its flaws, and it is not coextensive with relational databases. C.J. Date and Hugh Darwen write at very great length about this.
some other unnamed database "pioneer"
10:1 he's talking about Jim Starkey...
Btw, Domas helped set up Wikipedia. He's got more than FB engineering to his credit
"Scaling data systems in real life has humbled me. I would not dare criticize an architecture that the holds social graphs of 750M and works"
You know, you can't really argue with that.
Both Facebook, Google and Amazon deal with enormous numbers of data and users, and break exceedingly rarely.
Apparent failure vs failure
"Both Facebook, Google and Amazon deal with enormous numbers of data and users, and break exceedingly rarely."
They *apparently* break exceedingly rarely. Because the DBs (at least) are cleverly sharded, individual shard failures only bring down the system for a small number of users whose data resides on that shard. These individual failures are probably very common, but they don't really matter in the grand scheme of things.
Works up to a point.
Above that point you end up having to do so much genuflecting (replication, memcached, etc etc) to keep it running that it's worth looking at replacing the backend with something else - which may work a lot better with fewer resources.
Using it on something the size of FB is a recipe for pain but I understand the inertia involved. The question is if it's cheaper to keep making MySQL work or expend the manhours replacing it with something else. My bet is that right now they're so busy firefighting that they're afraid to even think about alternative backends.
I look forward to the thrilling arms race between Google and Facebook as they attempt to create the largest dataquasar in history. Fuelled entirely by kittypix..
- Fee fie Firefox: Mozilla's lawyers probe Dell over browser install charge
- Did Apple's iOS make you physically SICK? Try swallowing version 7.1
- Pics Indestructible Death Stars blow up planets using glowing KILL RAY
- Video Snowden: You can't trust SPOOKS with your DATA
- Review Distro diaspora: Four flavours of Ubuntu unpacked