back to article Google blames Gmail outage on data centre collapse

Google has apologised for yesterday’s major Gmail meltdown after some of its data centres in Europe failed to cope with a routine maintenance event. The company’s web-based email service was flat as a pancake for about three hours yesterday morning as Mountain View engineers attempted to fix the problem, which affected vast …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    Coat

    Oh....

    "Unexpected side effects of some new code"

    hehe

    Uh seriously though, 'Bad Times' we call that round these parts.

  2. Anonymous Coward
    Happy

    What no beta?

    They obviously didn't beta the code change long enough ;)

  3. Anonymous Coward
    Thumb Down

    99.9 Uptime?

    what ever happened to the holy grail of 5 nines? Or are we taking a step back these days and accepting a poorer service because it's on the hype cloud?

  4. Adam
    Happy

    Well Done

    Good work google keep it up!

  5. Anonymous Coward
    Anonymous Coward

    Single point of failure

    For all the redundancy in their networks and architecture, this has obviously exposed a single point of failure somewhere in the process.

    I have sympathy for them though. These kind of cascading failures are difficult to predict and Google are victims of their own success to an extent. Google apps is a great product and if Microsoft had anything like this, we would be used to having these kind of outages every few days rather than every few months (any big corporates switched to hotmail? no - I thought not)

  6. Andy ORourke
    Happy

    A FREE service

    "Many complained on the Google Apps and Gmail forums about the lengthy outage"

    Why complain, its a free email service, not life and death, I mean it's not as if anyone uses these free email services and online applications for anything important is it?

  7. Anonymous Coward
    Anonymous Coward

    What about us free users?

    Are they going to provide any compensation for us free users for the amount of hassle it has caused our businesses?

  8. James

    so does that mean...

    that all the google employees were logging into their yahoo!/hotmail accounts to get information?

  9. Anonymous Coward
    Anonymous Coward

    love the pun in -

    "Good work google keep it up!"

    or was that not meant to be a pun?

  10. Joseph Haig

    Re 99.9 Uptime?

    5 nines or 9 fives?

  11. Anonymous Coward
    Thumb Down

    45 minutes a month?

    The only time we've had that happen in years is when we're doing a major, planned upgrade. Unplanned, it's probably been 5 years. Most months we have between zero and two minutes of downtime. And we're not spending much.

    45 minutes a month? That's 540 minutes over the course of the year. 9 hours? Really? I feel like Google ought to be doing better than I am.

  12. Frank Bough
    Thumb Up

    The Interwhat?

    "usinesses and individuals were unable to gain access to their email accounts via the internet until around 12:30 GMT on Shrove Tuesday." So my completely unaffected IMAP connection to my gmail account was not performed via the internet? Spooky.

  13. floweracre
    Paris Hilton

    European Farms

    Flipping French Farmers

  14. Doug Southworth
    Stop

    @ 45 minutes a month?

    I'm going to go out on a limb and assume that you don't have 113 million users either...or datacenters on other continents for that matter. Things happen. Stuff goes down. Besides, Google isn't saying that they are going to use their 45 minutes of down time every month, just that they are allowed it.

    Seems that everyone likes to see the giant stumble, but no one would want his job on a day to day basis.

  15. Anonymous Coward
    Anonymous Coward

    compensation for free users?

    You get what you pay for.

  16. regadpellagru
    Paris Hilton

    @What about us free users

    "Are they going to provide any compensation for us free users for the amount of hassle it has caused our businesses?"

    Yes, you're provided 2 additionnal months of free service.

    *shakes head* People have a great service at 0 cost and they complain for a couple of hours of outage ! Try Orange webmail if you like !

    PH icon obviously, for clueless.

  17. Robin

    @A FREE service

    Some people are paying for this service

    I would have thought maybe google would have two systems running. One for the free users and one for the paid for users. Then they could try out all their changes on the free users first...

    Also interesting how when Google suffer an outage the public reaction is like: "no worries, these things happen", but when it's anyone else (eg Microsoft or even Yahoo) it's "Bunch of losers, waste of space, Kill Bill".

  18. Dale Richards
    Stop

    @Compensation for free users?

    The free service has no SLA, so no, stop being silly.

    The "premier" service does have an SLA, however:

    http://www.google.com/apps/intl/en/business/details.html

  19. Anonymous Coward
    Linux

    @Robin

    "Also interesting how when Google suffer an outage the public reaction is like: "no worries, these things happen", but when it's anyone else (eg Microsoft or even Yahoo) it's "Bunch of losers, waste of space, Kill Bill".

    ...Remember the last time Windows crashed?

    ...Remember the last time google.com was down?

  20. Anonymous Coward
    Thumb Down

    Re: Single point of failure

    "if Microsoft had anything like this, we would be used to having these kind of outages every few days"

    I think you'll find Microsoft DO have something like this, called Outlook Live. In all the time we've used it, it's been rock solid. It's based on the next version of Exchange, and business will be able to buy it too, this summer.

    And our on-premise user Exchange 2007 system has had about a minute of downtime combined (all planned 15 second failovers) since April 2007.

    I think any business that switches to Google Apps is going to realise within the year that they've made a very silly decision!

  21. Wyrmhole

    Outage?

    So am I the only one that experienced no outage? Or did it not affect POP3?

  22. John A Fotheringham
    Unhappy

    Giving up gmail for Lent?

    <shudder>

  23. Anonymous Coward
    Stop

    Fail and Google

    It's interesting listening to all the freetards saying, "I would like to see Microsoft do better etc." Well, they do a lot better. On my corporate exchange server I haven't seen a minute of downtime in 4 years. Also, if the exchange server did go down for a short while, I would have my local cache of current e-mails in Outlook (so long as I didn't move to another machine). The problem with cloud services is that not only do you lose new e-mails when the service goes down, you also lose access to archived mails as well. I know gears is supposed to help resolve this, but in reality if you start designing all your cloud apps with extensive client side caching you are essentially back to the old model. People are quickly going to realise that cloud = fail.

    And a quick comment for Doug Southworth. It really doesn't matter how many people use your service. At the end of the day it is each user's personal experience (or collectively a corporate experience) that will decide whether the service is worthwhile or not. Google tout cloud services as being more resilient than local services. In reality cloud services just change the type of fail that you suffer. In my corporate environment if my local machine fails, I can just step to another computer and work from there. If all my apps were in the cloud and the cloud fails I'm shafted.

  24. Bryan B
    Stop

    The Web is not the Internet

    "Businesses and individuals were unable to gain access to their email accounts via the internet"

    Not true.

    There was no access via the WEB, most accounts were still accessible via the Internet using IMAP or POP3.

  25. John Liccione

    CTO, Cemaphore Systems

    As CTO of Cemaphore Systems and 19-year background in high-availability messaging and disaster recovery systems, I think that Google handled the outage in a professional manner—acting openly and swiftly to address the issue.

    These recent outages do highlight the critical yet often forgotten need for email continuity to complement the online email service provider offering. For example, Cemaphore specializes in software which syncs and protects email real time in the cloud. With our MailShadow product (for Gmail/ Google Apps or Microsoft Exchange Online), companies can replicate all their email, contact and calendar data between Exchange/Outlook and their hosted email solution with Google or Microsoft---or even between 2 cloud provider email systems. So if an email outage occurs, users can continue running. Check us out at www.cemaphore.com.

  26. Gordon Grant
    Thumb Up

    Outages are to be expected

    Heck I mean I do get the occational disconnect from google mail but that's normally due to me downloading an legal iso via torrent and that sucks up all the bandwidth leaving none for gmail. As a web interface based mail client it's not bad, I see Hotmail has taken on board some of the points from gmail like "mark this e-mail as read" which wasn't there before.

    I seem to remember way, way back the whole of the AT&T network failing because of 1 line of code caused a cascade failure of the whole network..

    Shit happens, the guys at Mountain View got stuck in, found the problem and fixed it as soon as they could.

    When things run fine everything in the garden is lovely, yet when something goes wrong everyone gets the knives out..

    1) If it's free don't complain

    2) If you paid for it, this outage bites just a bit but it was just the web face that went down.

    I do wonder how well an Exchange server in the open, i.e tuck away in the Head Office with everyone connected into it via LAN / WAN / VPN would hold up genuinely.

    Sure G-mail is behind a firewall but in comparasion they are like a Bank Vault where as most Exchange servers are fort knox, not accessable unless you have several keys, right IP, your IP is on the "okay to allow through list" on the firewall, your account has remote access allowed on it, your username is right, your password is right and yeah it's not "3pm on a wet tuesday afternoon" (some random event)..

    I've seen worse web mail interfaces,

    Yahoo, hotmail - they've improved

    Orange - thank god I've got POP..

  27. phil

    isp with exchange

    there are plenty of exchange setups at ISPs around the world.

    several in europe and at least one in the uk, its a different beast entirely to corporate exchange setups but do the job very well.

This topic is closed for new posts.

Other stories you might like