Google's worldwide data center network spans about 900,000 servers, according to an estimate based on new information the company has deigned to share about its power use. Previous estimates put Google's server count at over one million. In a new report from Stanford professor Jonathan Koomey on data center energy usage, Google …
I'm curious as to how often their hardware needs replacing, not from a failure perspective, but from an upgrade standpoint as that is a shitload of hardware in anybody's book.
Mostly dies before upgrade due
There's a video in the ether somewhere, in it a Google boffin explains that they generally run low spec commodity hardware, and run it hard, just like a frenchman driving a Citroen.
This means that the hardware doesn't last very long at all, so is replaced long before an upgrade would be due, so they keep up to date due to natural wastage.
Google did a survey on hard drives as well that most hard drive manufacturers have devoured to see how their drives cope with extremely high workloads. In one data-farm Google were losing a drive every 45 minutes or so.
PDF report - http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en//papers/disk_failures.pdf
It is all I can think of.
Skynet is the one controlling all those shiny new Foxconn robots.
Are they hardened against EMP?
Operate without external power?
Require no human interference?
Skynet is a Virus!
Plus Google needs people to be alive for their model to work.
... you mean "Google+ needs people to be alive for their model to work" ?
Has anyone ever met anyone who works for Google? Has anyone ever been to their campus. Can anyone confirm that there are any humans at the company? Video of humans doesn't count. That can be faked. Google is Sky net.
re erm: "Plus Google needs people to be alive for their model to work."
The Matrix then, perhaps?
900,000 too low for 198MW of power.
I don't know who came out with this number of servers, probably not the Stanford professor, even the original article doesn't make that very clear. Either way it doesn't make sense.
198 Megawatt / 900,000 server = 220 Watts per server. That's a very high value for modern hardware running headless.
Even Quad core mobile processors plus discs, chipset, gigabit ethernet etc would work just fine with just 85W which would put Google's server number past the 2.3 million mark.
If they don't use mobile processors (which I think they do) the desktop versions would just be around 20W more, making the number of servers still double the prediction, at about 1.8 million.
Did you count A/C?
IIRC, data centers tend to spend half of the power they use in cooling.
RE: 900,000 too low for 198MW of power
I'd say it's a bit low actually. For a start, they are not laptops, they are servers.
Each Ethernet connection in the machine, and there will be at least 2 if not 4 or 6 will use 5W of power at 1Gbps, so potentially you have 30W just for the networking. Then add in 90W for a processor, a bit for mother board and so on and you're getting pretty high.
Google won't use hard drives in the servers either, they will boot from SAN, or PXE boot from a SAN connected server, but the thousands of hard drives will still use a lot of power.
In our 2KW racks in London, we can fit around 3 servers (HP DL380) before going over power/cooling restrictions. I expect Google's systems are more efficient than that but I doubt they are bellow 220W each.
..for doing maths?
Data centres use power for things other than servers. Cooling/lighting/security systems etc
~200 watts per server seems about right
the lowest-end 1u rack server from dell has a 250w power supply. which might be over-specced to provide enough headroom for power-hungry options - more disk, faster cpu, etc.
fyi the last one i installed in a data centre pulled down 0.8 amps @ 220 volts. i was there when this was checked 'cos the data centre's pricing depended in part on the amount of power used.
so around 200w for a server seems about right as a rough guide.
remember too that a rack full of commodity servers concentrates a *lot* of heat in a small space: about 10kw in half a cubic metre. that will need a fair bit of power for cooling.
so even if google could get its servers to just eat say 100w, there would still be about the same again for fans and aircon.
Downvoted for doing bad maths. He failed to account for the energy expended cooling 900k servers.
...but at the time I commented there was just a downvote, no explanation.
Interesting discussion, thanks guys/gals. Yes I did fail to account for cooling, although I suspect Google's unique enclosures and now chiller-less data centres brings that down quite a bit from the figures people are used to.
Google also apparently doesn't run SANs, they have local hard drives in each node and possibly share everything using their distributed file system. At least that's what I can tell from their video:
http://www.youtube.com/watch?v=1SCZzgfdTBo (at 0:50 for shots of some sort of server they run)
Looking closely at that video it seems they are running a dual CPU setup, so those would be server CPUs then. Maybe the 900,000 servers is right, accounting for 4 or 8 cores per server.
In my view 900,000 lowly dual cores would be too little to support their huge number of concurrent users and activities, that's why I was originally suspicious. But 900,000 packing more CPU power would fit the bill.
RE: 900,000 too low for 198MW of power
Google uses no SAN or NAS but use their 2 build in normal desktop (maybe laptop nowadays) sata disk to form their ever expanding uber GFS storage thing. Network interaces tops probable at 2 certainly no more (give me one good reason). The only thing they spend some money on is on ECC memory, all the rest they buy the best bung for the buck. So have a look in the price list of the most common HW vedors and you can make a good guess what a Google server could look like.
Yeah. I've noticed a lot of weird downvoting lately.
cooled solely with water from the Baltic Sea...
OMG! Google is dumping its unwanted data into the Baltic! This is bound to have an unprecedented effect on the ecology of the region! Is anyone from Greenpeace or Friends of the Earth reading this?
It's already a problem
See the paper of record - The Onion:
how much energy is required to pump that water ?
How much longer before governments start levelling an environmental tax on Internet connections? The more data you transfer, the more you pay because you're causing more servers to burn more juice.
I really could see them doing that before long... whilst letting big businesses continue polluting the environment and burning oil as though there's a big cash prize for whoever makes it run out the fastest.
You'll only encourage the fuckers.
> as though there's a big cash prize for whoever
> makes it run out the fastest.
Isn't that exactly what the stock market does?
Governments already tax the burning of more juice as they tax the electricity being used - a much more sensible place to put the taxation as it rewards more efficient data centres.
They impose both direct taxes on it and also indirectly through the requirements of electricity providers buying a minimum amount of green energy (pushing prices up and encouraging furthe development)
The UK Government already do. There are two key taxes levied against Datacentre and connectivity providers:
Fibre Tax - A tax on the physical fibre in the ground
Carbon Reduction Commitment - A tax on power usage
"Governments already tax the burning of more juice"
Only in their own country. Levelling a data tax would mean every time you download something from a foreign server, your local government tax you for that as well! And that's on top of the VAT on your electricity bill.
I really must suggest this one to the Minister later on today.
Sir Humphrey Appleby, GCB, KBE, MVO, MA (Oxon)
Working out the power
So if we take the 220MW and run an average PUE of 1.25 or 1.2 you get between 176 and 183MW of IT load.
With 900,000 servers @ 150ish Watts you have approx 135MW, the extra 40-48MW could easily be Networking and Storage for those 900,000 servers.
That much horsepower is so geekly awesome, it can only make you smile. Don't analyze, just bask, and wear your Faraday cod-piece.
Can you say "margin of error"?
"Google estimates that it accounts for less than one per cent of the energy consumed across the world's data centers, and Koomey uses this figure to extrapolate Google's worldwide server count. "
So in other words, a figure has been plucked out of the air, and then someone else has plucked another one out, thinking it may be related.
No need for "in San Francisco"
When your world view is "California to Finland to Asia".
Don't forget you also have to cool the datacentre, light the offices and provide facilities for the staff. All of which reduces the amount of energy left over for the servers themselves.
quality stuff, reminds me of Facebook 9000 mySQL database servers to run facebook and another 50,000 for the webservers. They don't seem to be very efficient are they, just throw hardware at the problem and hope will go away. Will be interested to know what other companies like microsoft, yahoo are using.
...this may give little insight into Yahoo.
I had to go for 1) interview and 2) test to see if I qualify for the next Finnish course.
Had an hour between the former and the latter.
Went to stretch my legs, and jigger me, there's a Google office. Oulu, Finland??? (Basically the back-of-beyond).
Not so back of...
Oulu used to be almost a Nokia company town (as you know). Don't know how much is left of Nokia there, but Oulu still has plenty of technology industry (and ex-Nokians). Google being there is not so surprising. Easier getting developers there than in Summa, where their Baltic-cooled data center is located.
There's a trend ...
... Away from big monolithic traditional servers to lots and lots of low-power low-spec processors (Atoms or ARM devices) in arrays.
A friend of mine saved his company many tens of thousands per annum by ditching their under-utilised trad Windows servers (urk) to use a small rack of MB's using low-powered Atoms paralleled up and running a distributed Linux. Works beautifully and costs next-to-nothing to run. Its also very fast at file serving. It runs very cool and sips electricity like a little old lady sipping tea.
Small is beautiful.
Long may it last
You think the Windows machines are bad, you should see how opaque the mainframe business is in this regard. Big Data users used to like to point to how they could get the maximum workload out of their gear by running at top speed all of the time, but with virtualisation, it is becoming possible for everyone to do that, to some extent, without paying out hand-over-fist for it.
(Of course the mianframe business is canny, and what will happen, is that IBM will take all of your cheap commodity servers and stick them in one big, chest-freezer-sized box, spray paint it mat black and sell it as a "Z1000", for a smidgen under a million dollars, with a label on the front saying "also runs Linux".)
900k is a lot of servers. I remember when AltaVista was the defacto search engine running on just 8 machines with 32GB RAM shared between them. Now you can have twice as much memory in a mobile phone. Amazing.