back to article Google's crash canaries' muted chirping led to load balancer brownout

Google has revealed that it broke its own cloud again, this time because of two failures: a software error and alerts that proved too hard to interpret. The problem hit Google's cloudy load balancers on Thursday, October 13, causing them to produce HTTP 502 (Bad Gateway) responses. At first, two per cent reported the problem. …

  1. Ole Juul

    even Google?

    So even Google can't keep the cloud from breaking? This does not bode well.

  2. Anonymous Coward
    Anonymous Coward

    It's indicative of the way the cloudy world has gone, but stories like this barely register more than a shrug these days.

    Oh look, a big, complex, multi-tenant platform has experienced an outage caused by a change. Meh.

  3. nematoad
    Unhappy

    I see.

    "“In this case, the change was pushed too quickly for accurate detection..."

    Looks like the "testing" stage was more of a box ticking exercise than actual testing. Either that or the testers were busy with other things and not watching what was actually going on.

  4. Gene Cash Silver badge
    Coat

    Obviously they need Yelling Bird... but that would probably be questionable content.

    1. Trevor_Pott Gold badge

      Or a really bad case of butts disease.

  5. Anonymous Coward
    Anonymous Coward

    “We typically have a period of soak time during a canary release"

    Google waterboard canaries? :-(

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon