Yahoo! looks beyond Google's data cruncher

El Reg HPC community – Where business meets supercomputing

Nowadays, when it comes to crunching epic amounts of web data, Google's MapReduce credo is all the rage. The Mountain View method of distributing back-end compute tasks across a sea of commodity machines has given rise to the open source Hadoop platform, which now underpins Yahoo!, Facebook, and even a chunk of Microsoft Bing. …

This topic is closed for new posts.

Posted Wednesday 17th February 2010 06:37 GMT

Bob 18

Back to the Future...

The idea that MapReduce might not be the be-all and end-all of parallel computing shouldn't be surprising. Consider the more "traditional" field of large-scale scientific computing (i.e. solving PDEs). MapReduce is absolutely NOT the fundamental way to get parallelism because the different parallel parts need to communicate on a fine-grained scale.

The surprise to me was that something as simple as MapReduce was able to solve AS MANY problems as it has.

Posted Wednesday 17th February 2010 15:46 GMT

Mr Young

I'm not sure?

So does this mean Yahoo! considers my web connection a distributed resource? Maybe I should send them I bill for using my leccy and PC? Hey - what about the cost of lighting, heating, beer, CPU cooling,coffees,biscuits? OMG - spreadsheet required...

Posted Thursday 18th February 2010 12:37 GMT

DZ-Jay

Re: Back to the Future...

@Bob 18:

That shouldn't really be surprising. As they say, when all you have is a hammer, everything looks like a nail.

Ever since MapReduce was described in that seminal white-paper, it became all the rage, and a lot of people did see it as the end-all, be all or distributed processing; and I posit that *because* of that, a lot of people have replaced their entire toolbox with this single hammer.

So then, it's not that MapReduce is able to solve so many problems, it's that it's been applied to so many problems, irrespective of its disposition to solve such problems.

-dZ.

This topic is closed for new posts.