How soon we forget our history
There is nothing (historically) new here. Back in the dark days of the 1970's, AT&T built UNIX for essentially the same reason: they needed LOTS and LOTS of switches, and figured it was cheaper to develop a small, portable operating system that could be used to create a switch from any piece of hardware. AT&T went on to develope the "B" series of computers (1B, 3B, 5B, 10B and 15B) to allow them to deploy any size switch anywhere they wanted, and put UNIX on all of them. That UNIX also worked as a general purpose OS was a side benefit (part of the "D" in "R&D"), but the "B" series was serious "iron" in its day.
When I worked in banking during the 1980's many large banks did the same thing. At Security Pacific we had purchased a mish-mash of hardware from General Automation, Control Data and Interdata and built our own custom branch automation system, including our own highly customized version of the GA CONTROL operating system. MUCH cheaper, faster, and most important, USEFUL than any solution from IBM, Siemens, NCR or other "turn-key" vendor.
While Google's work is impressive by any standard, we in the IT community need to keep in mind what they do and WHY they need to do it that way. Custom technology ALWAYS has a place in the extreem limits of any human endeavor. We don't think twice about space craft having custom computer and sensor systems, nor do we think twice about custom-built construction equipment for road building or skyscraper construction. We don't even consider all the "custom" computers that are embedded in cars, trucks, dishwashers, microwave or cell phones - again, something that can't be done with "COTS" (Common Off The Shelf) solutions.
The only thing that makes Google catch our eye is the scale: whole DATA CENTERS that are composed of custom hardware for a single purpose. Yet, to my "old guy" mind, this is nothing I've not already worked with nearly a dozen times before.
The most important lesson we can learn from Google's endeavor is how Google thinks: if the box won't work, think up a solution outside of it.
(I should stop here...but...)
Our biggest problem today is trying to force solutions to our problems to fit into nice, neat "boxes" of products that we can conveniently buy off the shelf at Wal*Mart. (Or HP. Or IBM. etc.) Most companies won't even consider an idea that doesn't have more vendor logos on it than a NASCAR racer. Build it in-house, for one purpose? You've got to be CRAZY!
And in most cases it's NOT necessary. But, if you do your planning and design right, and you DON'T see a solution that fits, and the prize is big enough...anybody got a hammer to help me break this box?