I hate this meme, really I do...
"Twitter takes to long to update!" ++spilt milk and screaming ADHD is what I get as a rational for the billions that will be pissed away on this.
While that (sadly) may be commercially viable enough of a reason to invest in "being able to process all the datas in a hadoop cluster in real time," I am really scratching my head trying to find a single useful application of the technology.
Hadoop batching and the like today gets us the ability to comb those structures somewhere around the 15 minute mark for largish datasets. So your news webpage would be 15 minutes behind. Egads!
15 minute lag time isn't going to be the end of the world for medical storage, geographical data storage, astronomical data, particle physics data, or anything else I can come up with. In fact, there are only two things I can think of that might be non-ADHD related that this innovation might enable.
Predicting earthquakes (highly unlikely, even with the proposed technological improvements outlines here,) and tracking everyone, everywhere, in real time. Not just online, but using image recognition, GPS, voice transmissions, etc. Sifting the mountains of “publicly available” information in real time to track a few billion people, find out what they buy, who they interact with, what they believe, how they vote and so forth.
Are we really proposing handing that technology over to our ever increasingly paranoid and unscrupulous governments in exchange for a quicker goddamned Twitter update?
What hath our obsession instant gratification wrought?