I'm curious as to how often their hardware needs replacing, not from a failure perspective, but from an upgrade standpoint as that is a shitload of hardware in anybody's book.
Google's worldwide data center network spans about 900,000 servers, according to an estimate based on new information the company has deigned to share about its power use. Previous estimates put Google's server count at over one million. In a new report from Stanford professor Jonathan Koomey on data center energy usage, …
There's a video in the ether somewhere, in it a Google boffin explains that they generally run low spec commodity hardware, and run it hard, just like a frenchman driving a Citroen.
This means that the hardware doesn't last very long at all, so is replaced long before an upgrade would be due, so they keep up to date due to natural wastage.
Google did a survey on hard drives as well that most hard drive manufacturers have devoured to see how their drives cope with extremely high workloads. In one data-farm Google were losing a drive every 45 minutes or so.
PDF report - http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en//papers/disk_failures.pdf
I don't know who came out with this number of servers, probably not the Stanford professor, even the original article doesn't make that very clear. Either way it doesn't make sense.
198 Megawatt / 900,000 server = 220 Watts per server. That's a very high value for modern hardware running headless.
Even Quad core mobile processors plus discs, chipset, gigabit ethernet etc would work just fine with just 85W which would put Google's server number past the 2.3 million mark.
If they don't use mobile processors (which I think they do) the desktop versions would just be around 20W more, making the number of servers still double the prediction, at about 1.8 million.
I'd say it's a bit low actually. For a start, they are not laptops, they are servers.
Each Ethernet connection in the machine, and there will be at least 2 if not 4 or 6 will use 5W of power at 1Gbps, so potentially you have 30W just for the networking. Then add in 90W for a processor, a bit for mother board and so on and you're getting pretty high.
Google won't use hard drives in the servers either, they will boot from SAN, or PXE boot from a SAN connected server, but the thousands of hard drives will still use a lot of power.
In our 2KW racks in London, we can fit around 3 servers (HP DL380) before going over power/cooling restrictions. I expect Google's systems are more efficient than that but I doubt they are bellow 220W each.
Google uses no SAN or NAS but use their 2 build in normal desktop (maybe laptop nowadays) sata disk to form their ever expanding uber GFS storage thing. Network interaces tops probable at 2 certainly no more (give me one good reason). The only thing they spend some money on is on ECC memory, all the rest they buy the best bung for the buck. So have a look in the price list of the most common HW vedors and you can make a good guess what a Google server could look like.
Interesting discussion, thanks guys/gals. Yes I did fail to account for cooling, although I suspect Google's unique enclosures and now chiller-less data centres brings that down quite a bit from the figures people are used to.
Google also apparently doesn't run SANs, they have local hard drives in each node and possibly share everything using their distributed file system. At least that's what I can tell from their video:
http://www.youtube.com/watch?v=1SCZzgfdTBo (at 0:50 for shots of some sort of server they run)
Looking closely at that video it seems they are running a dual CPU setup, so those would be server CPUs then. Maybe the 900,000 servers is right, accounting for 4 or 8 cores per server.
In my view 900,000 lowly dual cores would be too little to support their huge number of concurrent users and activities, that's why I was originally suspicious. But 900,000 packing more CPU power would fit the bill.
the lowest-end 1u rack server from dell has a 250w power supply. which might be over-specced to provide enough headroom for power-hungry options - more disk, faster cpu, etc.
fyi the last one i installed in a data centre pulled down 0.8 amps @ 220 volts. i was there when this was checked 'cos the data centre's pricing depended in part on the amount of power used.
so around 200w for a server seems about right as a rough guide.
remember too that a rack full of commodity servers concentrates a *lot* of heat in a small space: about 10kw in half a cubic metre. that will need a fair bit of power for cooling.
so even if google could get its servers to just eat say 100w, there would still be about the same again for fans and aircon.
How much longer before governments start levelling an environmental tax on Internet connections? The more data you transfer, the more you pay because you're causing more servers to burn more juice.
I really could see them doing that before long... whilst letting big businesses continue polluting the environment and burning oil as though there's a big cash prize for whoever makes it run out the fastest.
Governments already tax the burning of more juice as they tax the electricity being used - a much more sensible place to put the taxation as it rewards more efficient data centres.
They impose both direct taxes on it and also indirectly through the requirements of electricity providers buying a minimum amount of green energy (pushing prices up and encouraging furthe development)
"Governments already tax the burning of more juice"
Only in their own country. Levelling a data tax would mean every time you download something from a foreign server, your local government tax you for that as well! And that's on top of the VAT on your electricity bill.
I really must suggest this one to the Minister later on today.
Sir Humphrey Appleby, GCB, KBE, MVO, MA (Oxon)
"Google estimates that it accounts for less than one per cent of the energy consumed across the world's data centers, and Koomey uses this figure to extrapolate Google's worldwide server count. "
So in other words, a figure has been plucked out of the air, and then someone else has plucked another one out, thinking it may be related.
quality stuff, reminds me of Facebook 9000 mySQL database servers to run facebook and another 50,000 for the webservers. They don't seem to be very efficient are they, just throw hardware at the problem and hope will go away. Will be interested to know what other companies like microsoft, yahoo are using.
Oulu used to be almost a Nokia company town (as you know). Don't know how much is left of Nokia there, but Oulu still has plenty of technology industry (and ex-Nokians). Google being there is not so surprising. Easier getting developers there than in Summa, where their Baltic-cooled data center is located.
... Away from big monolithic traditional servers to lots and lots of low-power low-spec processors (Atoms or ARM devices) in arrays.
A friend of mine saved his company many tens of thousands per annum by ditching their under-utilised trad Windows servers (urk) to use a small rack of MB's using low-powered Atoms paralleled up and running a distributed Linux. Works beautifully and costs next-to-nothing to run. Its also very fast at file serving. It runs very cool and sips electricity like a little old lady sipping tea.
Small is beautiful.
You think the Windows machines are bad, you should see how opaque the mainframe business is in this regard. Big Data users used to like to point to how they could get the maximum workload out of their gear by running at top speed all of the time, but with virtualisation, it is becoming possible for everyone to do that, to some extent, without paying out hand-over-fist for it.
(Of course the mianframe business is canny, and what will happen, is that IBM will take all of your cheap commodity servers and stick them in one big, chest-freezer-sized box, spray paint it mat black and sell it as a "Z1000", for a smidgen under a million dollars, with a label on the front saying "also runs Linux".)
Biting the hand that feeds IT © 1998–2019