Custom hardware, custom software, efficiency?
To go a bit further with this idea, if we say that hypothetically, all Google servers are iron - no VMs. Unlikely, but bear with me.
Linux on x86, even if you pare down the OS to the bare bones, will still waste power - for example, if you have a farm that runs at 100% all the time and uses no VM sets, then the power management and VT instruction sets are wasted silicon, and probably wasting power.
Cut those out, you might get a 2% power saving.
2% isn't much, but 2% of a massive power bill is still a large number.
And if they rewrite the OS that the servers are running for that specific task to use every last drop of the pared down silicon, then they get near maximum efficiency from the available silicon - so performance increases, as they can put transistors for floating point operations where previously power saving/speedstep etc sets were - meaning more FLOPs per mm2.
Or am I reading too much into this, and showing just how little I know about instruction-set level hardware ops?
Still, it's an interesting idea - and Google certainly have the money and the people to do something of that ilk - and the scale of operations to make it a viable option.
PS: No need to flame me if I am genuinely talking rubbish - it's just a thought...