Feeds

back to article Mighty WAN pumper offered in the struggle to cope with Big Data

Companies may be excited about doing Google-style analytics on all aspects of their business with Hadoop and other "big data" tools, but big businesses are bracing for bigger phone bills as big data is starting to generate big traffic across the distributed operations of enterprises. This is music to the ears of Infineta Systems …

COMMENTS

This topic is closed for new posts.

Sanity Checks

Two corrections to this article

1. Google do not run Hadoop internally. They have Google FS, BigTable, Pregel and other things. The Apache Hadoop stack is evolving to be equivalent, but Google have their own stack, which predates much of Hadoop. The paper gets this right; it's the El Reg journalists who appear confused.

2. Bandwidth after the "MapReduce" stage is normally much less than ingress bandwidth. Hint, the word "reduce". This usually means squeezing down log data and the like to smaller summary.

Regarding the ingress/egress bandwidth, if all you are collecting is internal log data, you can predict the data rate (your daily click count, compressed), and its origin (your servers). Click log bandwidth will always be much less than site bandwidth, unless your site is something like bit.ly that just bounces 302 redirects back to the caller, in which case it's probably equal. Provided you keep the web servers near the Hadoop cluster, the cluster ingress bandwidth will be straightforward to handle.

The paper looks specifically at the problem of "classic" enterprises (i.e. pre-web), where systems are widely distributed for historical reasons; intra-enterprise traffic becomes a problem. This is probably the case when the application is itself distributed (telcos, banks). if your servers are scattered across 20 datacentres for historical reasons, you should consolidate down for cost reasons.

Despite these critiques of the article, the paper itself is pretty good.

0
0
Anonymous Coward

subtitle....

"A pipe so fat it can satiate even the biggest boxes", oh yes, go on, how long did it take you to think up that smutty innuendo? Finish 2011 as you mean to go on... ;)

0
0
Silver badge

Finbar Saunders would be proud of that one.

0
0
Facepalm

Typos

Xylinx Vertex? I suspect that should be Xilinx Virtex

0
0
This topic is closed for new posts.