back to article 'Catastrophic' server disk-destroying glitch menaced Google cloud

Google has squashed a bug in its public cloud that threatened to accidentally delete users' virtual disks – an error so serious that one security researcher termed it potentially "catastrophic". The web king today admitted that one of its server migration commands could have "erroneously and permanently" deleted persistent …

COMMENTS

This topic is closed for new posts.
  1. poopypants

    Mixed feelings

    -1 for having a serious, potentially data destroying bug

    +1 for being open and honest about it, and keeping their customers informed

    1. JEDIDIAH
      Devil

      Re: Mixed feelings

      +1 for having this serious data destroying but right after they discounted their cloud storage prices.

  2. Destroy All Monsters Silver badge
    Trollface

    For the LULZ!

    "So I make an API call to move VMs w/ 'persistent' volumes and — depending on my command line client version — my VM is accidentally nuked?"

    Deal with it!

    1. swissrobin

      Re: For the LULZ!

      Pretty much agree with that.

      If this was your disk in your computer and some finger trouble, s/w glitch, or similar made the disk unusable, you would need to reach for your backups.

      If you didn't have backups, you would need to reach for your coat.

      Doing IT for real is a pain in the backside - which is why I don't do it :-)

  3. Anonymous Coward
    Anonymous Coward

    But aren't the physical hard drives backed up elsewhere, in case they go up in flames or something?

    so if any are deleted, they just pull them back off the backups. Right?

    I know it's cloud, so distributed and virtual, but at layer 1 there must still be some backups somewhere.

  4. Anonymous Coward
    Anonymous Coward

    Sometimes you pay a cheap price and you get what you pay for.

    If you put data in the cloud without at least two local backup copies on reliable (and physically separate) media, you are an idiot and should be fired with cause.

    1. Fatman

      RE: Sometimes you pay a cheap price and you get what you pay for.

      If you put data in the cloud without at least two local backup copies on reliable (and physically separate) media, you are an idiot and should be fired with cause taken out back, and shot.

      FTFY

    2. jnffarrell1

      Sometimes when you autodelete a movie

      you want delayed execution. Tough luck yesterday. Second chance tomorrow.

  5. petur

    Overreacting users?

    So, this bug has actually always been in there.... how many users were actually bitten by this?

    1. Destroy All Monsters Silver badge

      Re: Overreacting users?

      You got a downvote.

      This, I think, means that no-one knows, but some people get unreasonably excited whenever the Cloud is mentioned .... because ....

      THOSE WHO MOVE THEIR STUFF INTO THE CLOUD APPEASE HITLER IN MUNICH!

      1. Anonymous Coward
        Anonymous Coward

        Re: Overreacting users?

        > THOSE WHO MOVE THEIR STUFF INTO THE CLOUD APPEASE HITLER IN MUNICH!

        It is now to appease Putin in Moscow

        1. Destroy All Monsters Silver badge

          Re: Overreacting users?

          Hillary, please leave.

        2. Mombasa69

          Re: Overreacting users?

          Well said!

        3. Anonymous Coward
          Anonymous Coward

          Re: Overreacting users?

          "It is now to appease Putin in Moscow".

          That should now actually be

          "It is now to appease Tsar Vladimir Putin in Moscow"

  6. Anonymous Coward
    Anonymous Coward

    This makes no sense

    Google has who knows how many petabytes of unused disk space, why would they be aggressive about deleting stuff even if the user says "yes, I really really really want you to delete this"? Anything the user deletes, whether accidentally or deliberately, should not actually be unlinked and overwritten but rather simply hidden from the user until it has aged a month or so and a background garbage collection process can take care of deleting it "for real".

    If Apple and Microsoft have this figured out, with the trash can and recycler being better than nothing in this regard, why is Google courting disaster in this way? They found the bug and announced it, big deal. All software has bugs, even Google's, and it is unlikely that this was the only such bug in their software.

    1. h4rm0ny

      Re: This makes no sense

      Maybe it's my UNIX background, but when I tell a system to delete something, I mean delete it. (Not hide it away in some "trash can"). It's quite possible there might be confidential or legal reasons I want to know that the data really is deleted. Your approach may be acceptable for family photos or similar, but for business use I think it's best clients know exactly what has and hasn't happened with their data.

      1. DropBear
        Facepalm

        Re: This makes no sense

        when I tell a system to delete something, I mean delete it. ...right, because we all know once you hit "delete" all that data is just irreversibly and irrecoverably gone ...

        1. Charles Manning

          Re: This makes no sense

          ".right, because we all know once you hit "delete" all that data is just irreversibly and irrecoverably gone .."

          It depends very much on the file system(s) they are using as to whether the data is gone or not.

          They certainly are not running FAT with it's little undelete feature.

      2. Anonymous Coward
        Anonymous Coward

        Re: This makes no sense

        If you have "confidential or legal reasons" why something really needs to be deleted, why in the world would you be using cloud storage? You keep that stuff on your own servers, where you have control over it.

        1. The Godfather
          Pint

          Re: This makes no sense

          And where your own people can mess around with it...

      3. Scroticus Canis
        Facepalm

        Re: This makes no sense - @h4rm0ny

        If you overwrite the entire file's disk allocation with random garbage (8 times is good) and then do the same for each iteration of your backups where the file was ever stored you can then say the file is deleted. Otherwise it's just hard to get to.

        Got to agree with DropBear on that one.

        1. Bloakey1

          Re: This makes no sense - @h4rm0ny

          I am afraid I disagree with you old chap. Coercivity is a bit more complex than that and I have in the past recovered data from drives and storage media that were erased up to DOD standard.

          The best ways to clear data are :

          Degauss and use a hammer

          Encrypt and use a hammer

          Use a hammer

          Throw drive into the sea many miles off shore

          The only person I ever met that understood this was an allegedly semi literate Irish builder. He told me he needed a new hard drive for his machine. The nudge and wink indicated that his knowledge of computing was as good as mine.

          1. Anonymous Coward
            Anonymous Coward

            Re: DoD secure erase

            "I have in the past recovered data from drives and storage media that were erased up to DOD standard."

            The DoD erase (and similar) stuff may have been relevant forty years ago when a 2 megabyte disk drive occupied a foot of 19" rackspace, when the track spacing (on e.g. an RK05) was 100 tracks per inch, and the linear bit density was maybe two thousand bits per inch. You could almost read that kind of data with a layer of fine iron filings and a magnifying class (you could certainly see where the tracks were).

            How is DoD secure erase still relevant in the era of PRML recording and beyond? For the last ten years or more, disk drives have offered recording densities hundreds of gigabits per square inch. You can't go "off track" like you could forty years ago. You can't "partially erase" like you could forty years ago. Bit positions in a track move forward or backward depending on the data bits preceding and following them. There's simply no scope for a "remanent" field from the previous version of the track/sector.

            It's a miracle of engineering that modern drives work at all; the idea that some 1970s-origin techniques can be used for post-erase data retrieval is just laughable.

          2. Adam Reid

            Re: This makes no sense - @h4rm0ny

            "I have in the past recovered data from drives and storage media that were erased up to DOD standard"

            I have serious doubts about the veracity of this statement. Or to to be blunt, I do not believe you.

            This StackExchange thread has a good discussion on why this is not possible:

            http://skeptics.stackexchange.com/questions/13674/is-it-possible-to-recover-data-on-a-zeroed-hard-drive

            I can't find an example anywhere on the internet of anyone successfully recovering data from a drive that has been erased using a single pass of zeroes, never mind multiple passes.

      4. Mike Pellatt
        Facepalm

        Re: This makes no sense

        Maybe it's my Unix background, but I know that if i tell the system to delete something, the inode's listing will be removed from the directory I deleted from, But if the inode is listed in another directory somewhere else, the data will still be on the disk. And if the inode is open in a running process, the data will remain on the disk until the process exits or otherwise closes the inode.

    2. Charles Manning

      "why would they be aggressive about deleting stuff"

      Well for one reason, customer privacy.

      If a customer wants to run everything through the shredder, for whatever reason, then Google cannot hold onto copies to be viewed by the spooks or whatever.

      The customer has said to delete the data so Google must do it. End of story.

  7. Frank Marsh
    Facepalm

    Are these actions commutative?

    So when you move data, do you copy it and then delete the original, or is it the other way around? Does it really matter?

    Ouch. This doesn't seem like abject stupidity, it seems like a race condition that wasn't anticipated. So glad I wasn't the programmer.

    1. Anonymous Coward
      Anonymous Coward

      Re: presentation layer people

      "when you move data, do you copy it and then delete the original, or is it the other way around?"

      Surely you jest?

      "This doesn't seem like abject stupidity, it seems like a race condition that wasn't anticipated."

      No, it seems like abject clueless stupidity of the kind that even Microsoft would struggle to let out into the light of day.

      Oh, hang on... Mr Marsh's first post here:

      " I can say this confidently since I'm a patent attorney here in the U.S."

      The joys of having non-engineers do stuff that should be left to people with a clue, preferably with a clue about engineering.

    2. frank ly

      Re: Are these actions commutative?

      Apart from the word 'commutative' not really being applicable to copying and deleting data, the answer is "it depends". The simplest answer is that you change a pointer entry in a directory structure, if you're moving a file to be on the same logical hard drive on your home PC. After that, it can get quite complicated.

  8. Mombasa69

    I never trust cloud storage anyway, it's only there for the NSA and GCHQ to have easy access to your personal stuff and for Google to nose about into your private information anyway.

  9. Anonymous Coward
    Coat

    If they were...

    Running ZFS file system on a hardware biased raid 6 or 10? That would be a real laugh.

    Mines the one with the "File servers for complete novices" in the pocket. I'm on page 1, so would love to know what mistakes a professional company like Google make.

  10. willi0000000

    backups

    not to worry Google has long since entered into an agreement for provision of backups for it's entire system.

    [so if you lost a file, just ask NSA for a copy of the backup]

    1. Anonymous Coward
      Coffee/keyboard

      Re: backups

      Only if it was moved between Google data centers...

This topic is closed for new posts.

Other stories you might like