back to article Google cloud VMs given same IP addresses ... and down they went

Google gave some of its cloud customers a rotten weekend by breaking a bunch of virtual machines. Detailed in this incident report, the company first noticed problems at nearly beer o’clock on Friday afternoon, June 15th, Pacific Time – just after midnight on Saturday for European users and early Saturday morning in Asia. The …

  1. Yes Me Silver badge

    IPv what?

    Let's hope these were duplicate IPv4 addresses. Duplicate IPv6 would be unforgivable.

  2. This post has been deleted by its author

  3. dbtx Bronze badge

    Great, that's who's holding GitLab now. Who was it that said 'out of the frying pan' last week?

    @AC: "Cloud means Somebody Else's Problem" ftfy.

    1. Anonymous Coward
      Anonymous Coward

      AC deleted his post. Who's holding GitLab, @dbtx?

      1. Anonymous Coward
        Anonymous Coward

        GitLab is hosted on Azure cloud. Ironically, M$ has now full read access to private GitHub and GitLab repos.

  4. Anonymous Coward
    Anonymous Coward

    More Google CloudFog

    Its been said many times before on here.... Cloud means someone else's computer. But you can't deny Cloud convenience is great when everything is working... Its just a different kind of weekend call out, when things aren't...

    1. Scott Marshall

      Re: More Google CloudFog

      The call-out rate for Cloud-based services is determined by the "risk" multiplicand I call the "Cloud Fuctor".

      The more services that one has "in the Cloud", the higher the "Cloud Fuctor" multiplicand, which will always be ≥ 1.0

      The risk can be mitigated by distributing across multiple providers.

      1. Adam 52 Silver badge

        Re: More Google CloudFog

        I can see why you'd do that; obviously getting out of bed and driving to the data center is so much cheaper than logging into a console from bed.

      2. Sir Loin Of Beef

        Re: More Google CloudFog

        The risk can be mitigated by building and managing your own sh*t. Letting others manage your stuff just makes no sense.

        1. John Robson Silver badge

          Re: More Google CloudFog

          - The risk can be mitigated by building and managing your own sh*t. Letting others manage your stuff just makes no sense.

          Well that depends on how many bits of kit you would have to manage, and how many people you have available and competent to manage them.

          The risks involved in hosting your own are that you end up coming in on Monday to find everything offline because even your monitoring system has gone down. Then you have to figure out what has gone wrong where and drive round the country fixing bits...

          There are risks with cloud deployments, but those risks can be measured and compared with 'in house' risks... and the balance, for some/many may well be in favour of the cloud.

  5. MAH

    I am guessing it's not duplicate IP addresses but duplicate mac addresses.

    Keeping the boot disk bit recreating the configuration would have generated a new Mac address

    1. fronty

      Huh? The article said duplicate IP address, why do you think they meant duplicate MAC address?

      1. Anonymous Coward Silver badge
        Paris Hilton

        Because DHCP servers tend to key on the MAC address.

  6. Anonymous Coward
    Anonymous Coward

    This kind of incidents

    will slowly increase in number and will take longer to be solved. So far everything is under control because those maintaining and managing the cloud infrastructure are those brilliant minds who designed it and set it up. Come the next generation and the one after things will be in dire situation when those called to support this massive infrastructure will have to rely on documentation, good or bad. Yes, there will be automation with scripts running the show but when things will go wrong it will all become tricky.

    1. Grikath Silver badge

      Re: This kind of incidents

      Ahhhh... But by that time the Cloud is Old Hat, and the new hype will be "Decentralised Microstorage" or "Dedicated Localhosting" or somesuch. Which basically means the servers go back onsite, with a profitable remote-managing contract, of course....

    2. vtcodger Silver badge
      Unhappy

      Re: This kind of incidents

      <blockquote>will slowly increase in number and will take longer to be solved.</blockquote>

      Interesting point. There is presumably a "Fault Surface" similar or maybe identical to the malware "attack Surface" that expands as interfaces become more "flexible" and complex. Problem is that the intelligence of those managing the interfaces doesn't expand to match the increasing size of the Fault Surface.

      Back in the 1960s, as we discovered that implementing simple ideas on computers was anything but simple, we used to say the FLEX was a four letter. Brace yourself cloud-people, we are probably going to be flexed repeatedly in coming years.

    3. bombastic bob Silver badge

      'The Cloud' - highly overrated

      'Highly Overrated'©®�⍟☂☢™ since its invention around 2005 or 2006 (when AWS came out). Market-droids have been pushing it ever since.

  7. Kevin McMurtrie Silver badge

    Five nines of uptime

    92.9999%

    Three times a week: Network vanishes for a moment then comes back.

    Once a week: Slow for a few hours due to provisioning something-something.

    Once a month: An engineer makes a mistake and everything's dead on a Friday afternoon.

    1. Anonymous South African Coward Silver badge

      Re: Five nines of uptime

      Once a month: An engineer makes a mistake and everything's dead on a Friday afternoon.

      A law need to be passed that anything critical must not be done on a friday afternoon, only on a moanday morning (or, even better, tuesday morning).

      This will allow world+dog techies to collect their beer at pub o' clock and not having to waste their friday afternoons wrestling with issues some daft ******* caused in the first place.

  8. MatsSvensson

    Sharing economy

    Sharing economy 2.0

    1. Voland's right hand Silver badge

      Re: Sharing economy

      Close. Should be Sharing Economy v4.

      1. Giovani Tapini

        Re: Sharing economy

        must be twice as good as 2.0 then...

        1. robidy

          Re: Sharing economy

          Explains why MS went with years all of a sudden.

  9. handleoclast Silver badge
    Thumb Up

    Automation for the win!

    With servers as pets, you could only fuck them up one at a time, and you might be notified of the problems you'd caused before you fucked up many of them.

    With the automation tools for cloud servers you can fuck up the entire lot in one fell swoop. This is a fantastic productivity gain.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019