back to article Googlenet runs on '900,000 servers'

Google's worldwide data center network spans about 900,000 servers, according to an estimate based on new information the company has deigned to share about its power use. Previous estimates put Google's server count at over one million. In a new report from Stanford professor Jonathan Koomey on data center energy usage, …

COMMENTS

This topic is closed for new posts.
  1. Mark 65

    Replacement rate

    I'm curious as to how often their hardware needs replacing, not from a failure perspective, but from an upgrade standpoint as that is a shitload of hardware in anybody's book.

    1. Alan 6

      Mostly dies before upgrade due

      There's a video in the ether somewhere, in it a Google boffin explains that they generally run low spec commodity hardware, and run it hard, just like a frenchman driving a Citroen.

      This means that the hardware doesn't last very long at all, so is replaced long before an upgrade would be due, so they keep up to date due to natural wastage.

      Google did a survey on hard drives as well that most hard drive manufacturers have devoured to see how their drives cope with extremely high workloads. In one data-farm Google were losing a drive every 45 minutes or so.

      PDF report - http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en//papers/disk_failures.pdf

  2. Anonymous Coward
    Meh

    Skynet...

    It is all I can think of.

    1. Aichikenmin
      Angel

      Nonono

      Skynet is the one controlling all those shiny new Foxconn robots.

    2. Anonymous Coward
      Terminator

      Perhaps?

      Are they hardened against EMP?

      Operate without external power?

      Require no human interference?

      Oh.........

    3. Captain Scarlet Silver badge
      Paris Hilton

      erm

      Skynet is a Virus!

      Plus Google needs people to be alive for their model to work.

      1. I'm Brian and so's my wife
        Joke

        Shirley...

        ... you mean "Google+ needs people to be alive for their model to work" ?

      2. Gil Grissum
        Pint

        People?

        Has anyone ever met anyone who works for Google? Has anyone ever been to their campus. Can anyone confirm that there are any humans at the company? Video of humans doesn't count. That can be faked. Google is Sky net.

      3. ZweiBlumen
        Terminator

        Matrix?

        re erm: "Plus Google needs people to be alive for their model to work."

        The Matrix then, perhaps?

  3. Anonymous Coward
    Headmaster

    900,000 too low for 198MW of power.

    I don't know who came out with this number of servers, probably not the Stanford professor, even the original article doesn't make that very clear. Either way it doesn't make sense.

    198 Megawatt / 900,000 server = 220 Watts per server. That's a very high value for modern hardware running headless.

    Even Quad core mobile processors plus discs, chipset, gigabit ethernet etc would work just fine with just 85W which would put Google's server number past the 2.3 million mark.

    If they don't use mobile processors (which I think they do) the desktop versions would just be around 20W more, making the number of servers still double the prediction, at about 1.8 million.

    1. ratfox

      Did you count A/C?

      IIRC, data centers tend to spend half of the power they use in cooling.

    2. Lusty

      RE: 900,000 too low for 198MW of power

      I'd say it's a bit low actually. For a start, they are not laptops, they are servers.

      Each Ethernet connection in the machine, and there will be at least 2 if not 4 or 6 will use 5W of power at 1Gbps, so potentially you have 30W just for the networking. Then add in 90W for a processor, a bit for mother board and so on and you're getting pretty high.

      Google won't use hard drives in the servers either, they will boot from SAN, or PXE boot from a SAN connected server, but the thousands of hard drives will still use a lot of power.

      In our 2KW racks in London, we can fit around 3 servers (HP DL380) before going over power/cooling restrictions. I expect Google's systems are more efficient than that but I doubt they are bellow 220W each.

      1. blondie101
        Holmes

        RE: 900,000 too low for 198MW of power

        Google uses no SAN or NAS but use their 2 build in normal desktop (maybe laptop nowadays) sata disk to form their ever expanding uber GFS storage thing. Network interaces tops probable at 2 certainly no more (give me one good reason). The only thing they spend some money on is on ECC memory, all the rest they buy the best bung for the buck. So have a look in the price list of the most common HW vedors and you can make a good guess what a Google server could look like.

    3. sabroni Silver badge
      WTF?

      Downvoted..

      ..for doing maths?

      1. Tom 38
        Thumb Down

        @sabroni

        Downvoted for doing bad maths. He failed to account for the energy expended cooling 900k servers.

        1. sabroni Silver badge

          ok..

          ...but at the time I commented there was just a downvote, no explanation.

        2. Anonymous Coward
          Anonymous Coward

          Interesting

          Interesting discussion, thanks guys/gals. Yes I did fail to account for cooling, although I suspect Google's unique enclosures and now chiller-less data centres brings that down quite a bit from the figures people are used to.

          Google also apparently doesn't run SANs, they have local hard drives in each node and possibly share everything using their distributed file system. At least that's what I can tell from their video:

          http://www.youtube.com/watch?v=1SCZzgfdTBo (at 0:50 for shots of some sort of server they run)

          Looking closely at that video it seems they are running a dual CPU setup, so those would be server CPUs then. Maybe the 900,000 servers is right, accounting for 4 or 8 cores per server.

          In my view 900,000 lowly dual cores would be too little to support their huge number of concurrent users and activities, that's why I was originally suspicious. But 900,000 packing more CPU power would fit the bill.

      2. Hardcastle the ancient
        WTF?

        @sabroni

        Yeah. I've noticed a lot of weird downvoting lately.

    4. Anonymous Coward
      Boffin

      ~200 watts per server seems about right

      the lowest-end 1u rack server from dell has a 250w power supply. which might be over-specced to provide enough headroom for power-hungry options - more disk, faster cpu, etc.

      fyi the last one i installed in a data centre pulled down 0.8 amps @ 220 volts. i was there when this was checked 'cos the data centre's pricing depended in part on the amount of power used.

      so around 200w for a server seems about right as a rough guide.

      remember too that a rack full of commodity servers concentrates a *lot* of heat in a small space: about 10kw in half a cubic metre. that will need a fair bit of power for cooling.

      so even if google could get its servers to just eat say 100w, there would still be about the same again for fans and aircon.

    5. Andrew Lobban

      Remember

      Data centres use power for things other than servers. Cooling/lighting/security systems etc

  4. nyelvmark
    Alert

    cooled solely with water from the Baltic Sea...

    OMG! Google is dumping its unwanted data into the Baltic! This is bound to have an unprecedented effect on the ecology of the region! Is anyone from Greenpeace or Friends of the Earth reading this?

    1. deadlockvictim
      Big Brother

      It's already a problem

      See the paper of record - The Onion:

      http://www.theonion.com/articles/evil-genius-gates-drops-windows-98-into-nyc-water,557/

    2. GatesFanbois

      So...

      how much energy is required to pump that water ?

  5. Anonymous Coward
    Anonymous Coward

    Carbon footprint

    How much longer before governments start levelling an environmental tax on Internet connections? The more data you transfer, the more you pay because you're causing more servers to burn more juice.

    I really could see them doing that before long... whilst letting big businesses continue polluting the environment and burning oil as though there's a big cash prize for whoever makes it run out the fastest.

    1. CmdrX3
      Stop

      Shhhh!!!!!!!!!!

      You'll only encourage the fuckers.

    2. Hardcastle the ancient

      Carbon tax

      > as though there's a big cash prize for whoever

      > makes it run out the fastest.

      Isn't that exactly what the stock market does?

    3. The Mole

      Electricity

      Governments already tax the burning of more juice as they tax the electricity being used - a much more sensible place to put the taxation as it rewards more efficient data centres.

      They impose both direct taxes on it and also indirectly through the requirements of electricity providers buying a minimum amount of green energy (pushing prices up and encouraging furthe development)

      1. Anonymous Coward
        Anonymous Coward

        Re: Electricity

        "Governments already tax the burning of more juice"

        Only in their own country. Levelling a data tax would mean every time you download something from a foreign server, your local government tax you for that as well! And that's on top of the VAT on your electricity bill.

        I really must suggest this one to the Minister later on today.

        Kind regards,

        Sir Humphrey Appleby, GCB, KBE, MVO, MA (Oxon)

    4. James Smith 2

      Already happening

      The UK Government already do. There are two key taxes levied against Datacentre and connectivity providers:

      Fibre Tax - A tax on the physical fibre in the ground

      Carbon Reduction Commitment - A tax on power usage

  6. Michael Duke

    Working out the power

    So if we take the 220MW and run an average PUE of 1.25 or 1.2 you get between 176 and 183MW of IT load.

    With 900,000 servers @ 150ish Watts you have approx 135MW, the extra 40-48MW could easily be Networking and Storage for those 900,000 servers.

  7. J 37
    Happy

    Wicked cool!

    That much horsepower is so geekly awesome, it can only make you smile. Don't analyze, just bask, and wear your Faraday cod-piece.

  8. MrWibble
    Headmaster

    Can you say "margin of error"?

    "Google estimates that it accounts for less than one per cent of the energy consumed across the world's data centers, and Koomey uses this figure to extrapolate Google's worldwide server count. "

    So in other words, a figure has been plucked out of the air, and then someone else has plucked another one out, thinking it may be related.

  9. Anonymous Coward
    FAIL

    No need for "in San Francisco"

    When your world view is "California to Finland to Asia".

  10. BillboBaggins
    Headmaster

    @ +++ath0

    Don't forget you also have to cool the datacentre, light the offices and provide facilities for the staff. All of which reduces the amount of energy left over for the servers themselves.

  11. Anonymous Coward
    Anonymous Coward

    quality

    quality stuff, reminds me of Facebook 9000 mySQL database servers to run facebook and another 50,000 for the webservers. They don't seem to be very efficient are they, just throw hardware at the problem and hope will go away. Will be interested to know what other companies like microsoft, yahoo are using.

    1. Anonymous Coward
      Anonymous Coward

      Yahoo stuff...

      ...this may give little insight into Yahoo.

      http://www.theregister.co.uk/2010/09/23/yahoo_compute_coop/

  12. Andus McCoatover
    Windows

    Gobsmacked today...

    I had to go for 1) interview and 2) test to see if I qualify for the next Finnish course.

    Had an hour between the former and the latter.

    Went to stretch my legs, and jigger me, there's a Google office. Oulu, Finland??? (Basically the back-of-beyond).

    1. MacroRodent
      Holmes

      Not so back of...

      Oulu used to be almost a Nokia company town (as you know). Don't know how much is left of Nokia there, but Oulu still has plenty of technology industry (and ex-Nokians). Google being there is not so surprising. Easier getting developers there than in Summa, where their Baltic-cooled data center is located.

  13. This post has been deleted by its author

  14. Anonymous Coward
    Happy

    There's a trend ...

    ... Away from big monolithic traditional servers to lots and lots of low-power low-spec processors (Atoms or ARM devices) in arrays.

    A friend of mine saved his company many tens of thousands per annum by ditching their under-utilised trad Windows servers (urk) to use a small rack of MB's using low-powered Atoms paralleled up and running a distributed Linux. Works beautifully and costs next-to-nothing to run. Its also very fast at file serving. It runs very cool and sips electricity like a little old lady sipping tea.

    Small is beautiful.

    1. Daniel 1

      Long may it last

      You think the Windows machines are bad, you should see how opaque the mainframe business is in this regard. Big Data users used to like to point to how they could get the maximum workload out of their gear by running at top speed all of the time, but with virtualisation, it is becoming possible for everyone to do that, to some extent, without paying out hand-over-fist for it.

      (Of course the mianframe business is canny, and what will happen, is that IBM will take all of your cheap commodity servers and stick them in one big, chest-freezer-sized box, spray paint it mat black and sell it as a "Z1000", for a smidgen under a million dollars, with a label on the front saying "also runs Linux".)

  15. magnetik
    Happy

    huge

    900k is a lot of servers. I remember when AltaVista was the defacto search engine running on just 8 machines with 32GB RAM shared between them. Now you can have twice as much memory in a mobile phone. Amazing.

This topic is closed for new posts.

Other stories you might like