back to article Rookie almost wipes customer's entire inventory – unbeknownst to sysadmin

Welcome once again to El Reg’s weekly instalment of Who, Me?, where readers get monumental cock-ups and heart-stopping near-misses off their chests. This week, Reginald tells us a personal horror story from the ‘80s, when he worked for what was then a top five minicomputer biz. At the time, he was on site at a customer – a …

  1. big_D Silver badge

    Re: .cobol

    Yep, one of the many reasons I don't work there any more.

  2. DropBear Silver badge

    Re: .cobol

    "Delete is right above Rename in the bloody menu"

    Really. And you're not even going to try undeleting what is essentially a perfect candidate for recovery as long as you realise what you've done immediately...?

  3. big_D Silver badge

    Re: .cobol

    On a Samba share? No. No trashcan, no VSS.

  4. bpfh

    Re: .cobol

    "Delete is right above Rename in the bloody menu"

    Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

    I found this out - to my peril - about 3 seconds before I realised that it was a good idea for a server's crontab to include a daily executed crontab -l > /foo/bar/crontab-backup.txt ...

  5. Just An Engineer

    Re: .cobol

    "when the boss comes in at 5:30 in the evening and says he has a presentation at 10am at CeBIT the next day and needs 40 slides..."

    There are 4 little words that come to mind when this happens to me. "sucks to be you".

    Or conversely "it's really going to be a log night for you, boss."

    Usually when this happens the "Boss" has known about it for 3 to 4 weeks, then the Oh Shite moment happens and remembers the night before. Usually when his boss asks if he is ready for tomorrow...

  6. big_D Silver badge

    Re: .cobol

    He knew about it 6 months in advance. I'd told him 4 months in advance not to leave it to the last minute... He said he had everything under control.

  7. Jason Bloomberg Silver badge

    Re: .cobol

    I went to delete the original files, but I only got as far as "del *.COB" befiore hitting return.

    I managed a similar thing but more deliberately; belatedly finding "DEL FOOBAR.???" included files with no extensions when it didn't on a previous version (Win3.1?).

    That wasn't the disaster it could have been but I've had my share of all-nighters making it look like I hadn't accidentally scrubbed a system clean.

  8. Rich 11 Silver badge

    Re: .cobol

    He said he had everything under control.

    That's the warning sign, right there. Along with "Trust me, I know what I'm doing."

  9. schubb

    Re: .cobol

    I remember writing a wrapper to ensure I didn't do this very thing...

  10. Down not across

    Re: .cobol

    Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

    Using crontab -e is asking for trouble even without mistypes. I've see too many corrupted or truncated crontabs after someone has edited them with crontab -e. crontab -l > crontab.txt;vi crontab.txt;crontab crontab.txt is much better way.

    You mean not everyone has crontab entry that backs up crontab at least daily?

  11. MrBanana

    Re: .cobol

    "WAH! I copied the .COBOL back to .COB and started over again. As I knew what I wanted to do this time, it only took about a day to re-do what I had deleted."

    When this has happened to me, I end up with better code than I had before. Re-doing the work gives you a better perspective. Even if functionally no different it will be cleaner, well commented, and laid out more consistently. I sometimes now do it deliberately (although just saving the first new version, not deleting it) to clean up the code.

  12. big_D Silver badge

    Re: .cobol

    I totally agree, the resultant code was better than what I had previously written, because some of the mistakes and assumptions I'd made the first time round and worked around didn't make it into the new code.

  13. Blacklight

    Re: .cobol

    My SMB box has a .recycle

    Granted I have to enable it, but it's there - and on every family share "just in case" :)

  14. Woza

    Reminds me of the classic

    https://www.ee.ryerson.ca/~elf/hack/recovery.html

  15. Anonymous South African Coward Silver badge

    Re: Reminds me of the classic

    https://www.ee.ryerson.ca/~elf/hack/recovery.html

    Was about to post the same. It is a legendary classic by now.

  16. malle-herbert Silver badge
    Facepalm

    Try uninstalling a video game...

    Only to find out it wiped the entire drive !

    The game put it's saved games in the root of the drive...

    And after anwsering "yes" to the question "do you also want to delete all saved games" it was literally 'game over'...

  17. J.G.Harston Silver badge

    Re: Try uninstalling a video game...

    Not quite, but TEMP=C:\DOS

    Then install something that tidies up the TEMP directory on exit.....

  18. Chairman of the Bored Silver badge

    One simple trick...

    ...depending on your shell and its configuration a zero size file in each directory you care about called '-i' will force the rampaging recursive rm, mv, or whatever back into interactive mode. By and large it won't defend you against mistakes in a script, but its definitely saved me from myself when running an interactive shell.

    It's proven useful enough to earn its own cronjob that runs once a week and features a 'find -type d' and touch '-i' combo on systems I like.

    Glad the OP's mad dive for the power switch saved him, I wasn't so speedy once. Total bustification. Hence this one simple trick...

    Now if I could ever fdisk the right f$cking disk, I'd be set!

  19. Chris Evans

    Re: One simple trick...

    Can't you enter a command to abort the wipe?

    I'm no Unix expert as you will probable guess.

  20. PickledAardvark

    Re: One simple trick...

    "Can't you enter a command to abort the wipe?"

    Maybe. But you still have to work out what got deleted.

    On the first Unix system I used, an admin configured the rm command with a system alias so that rm required a confirmation. Annoying after a while but handy when learning.

    When you are reconfiguring a system, delete/rm is not the only option. Move/mv protects you from your errors. If the OS has no move/mv, then copy, verify before delete.

  21. Doctor Syntax Silver badge

    Re: One simple trick...

    "Move/mv protects you from your errors."

    Not entirely. I had a similar experience with mv. I was left with a running shell so could cd through the remains of the file system end list files with echo * but not repair it..

    Although we had the CDs (SCO) to reboot the system required a specific driver which wasn't included on the CDs and hadn't been provided by the vendor. It took most of a day before they emailed the correct driver to put on a floppy before I could reboot. After that it only took a few minutes to put everything back in place.

  22. Chairman of the Bored Silver badge

    Re: One simple trick...

    @Chris Evans,

    Yes there are a number of things you can do. Just like Windows a quick ctrl-C will abort a rm operation taking place in an interactive shell. Destroying the window in which the interactive shell running rm is running will work, too (alt-f4 in most window managers or 'x' out of the window)

    If you know the process id of the rm process you can 'kill $pid' or do a 'killall -KILL rm'

    Couple of problems:

    (1) law of maximum perversity says that the most important bits will be destroyed first in any accident sequence

    (2) by the time you realize the mistake there is no time to kill rm before law 1 is satisfied

    The OP's mad dive for the power button is probably the very best move... provided you are right there at the console. And provided the big red switch is actually connected to anything

  23. Julian 8

    Reminds me of a contractor

    We had in a company a few years ago. He wrote a routine to clear down a specific folder on a cluster. all fairly simple. I did not find out for 3 weeks, but over the last three weekends one of our team had been in fixing an issue with a cluster node that had failed.

    Eventually I was given the script and asked to look over and it all looked OK, nothing obvious, a few commands and a delete command. I then asked why they suspected this script and was told the node's boot drive had been wiped.

    The following weekend and again a node went down. I looked at the script again and ran it line by line instead of just looking and when it got to the delete line instead of wanting to delete the intended folder it wanted to delete the C:\ instead.

    Looked closer at the line and there was a trailing space after the folder name, so in those days Windows decided to erase the folder where it was (which due to how the task had been created it ran in c:\)

  24. Version 1.0 Silver badge
    Thumb Up

    Re: Reminds me of a contractor

    Windows was good at that - I had an employee rename win.ini for some reason back in the old days. When win.ini doesn't exists Windows reported that every single file on the system was called win.ini ... the funny (and best part of this story) was that she requested all the original system and application install disks because "she wanted to verify and understand how the system (our standard 'customers' installation) was setup and configured" ... I gave her all the disks and watched her rebuild the system over a couple of days.

    She learned a lot and because a damn fine customer support and installation engineer! There's always a silver lining to a cloud.

  25. JimC Silver badge

    One advantage of getting old

    Is that a dim mist has descended over most of my major cockups. I can remember a few of other people's though!

  26. The Man Who Fell To Earth Silver badge
    Unhappy

    Re: One advantage of getting old

    "I can remember a few of other people's though!"

    My (Irish) boss attributes this to "Irish Alzheimer's", which he says is "Long after the memory has gone, the grudge remains."

  27. Alien8n Silver badge

    New contractor

    We're currently breaking in a new contractor as the previous one has been forceably removed due to the complete donkey's arse of a server upgrade done here at work.

    Basically:

    Built new server.

    Transferred file from old to new.

    Kicked off backup routines.

    Copy remaining files over.

    Kill server.

    Except he made 3 rookie mistakes:

    1. He hadn't ensured all the shared folders had been copied across, as well as all the "secure" folders. So anyone with a locked down set of folders found all their files were missing.

    2. When doing the second transfer he used Robocopy. Not a problem except he was using copy/paste scripts and managed to include the /mir flag. Result being deleting the last 2 days of updates on the new file server. Not a problem, except for error 3.

    3. He kicked of the final Robocopy without first checking to make sure that the new backups had actually worked, meaning we had no backup of the last 2 day's work.

    In good news we'd made the decision prior to the server upgrade to leave the old SQL server in place until after we'd tested the new application server, so all that was lost was a handful of Excel and Word documents. In bad news (for him) he'd managed to cock up at several clients resulting in a nice shiny new P45.

  28. Anonymous South African Coward Silver badge

    Re: New contractor

    ....resulting in a nice shiny new P45.

    I have to assume that a P45 is a civil court case?

  29. WonkoTheSane
    Headmaster

    Re: New contractor

    "P45" is the UK government standard form used when one is invited to seek alternate employment.

  30. Anonymous South African Coward Silver badge

    Re: New contractor

    "P45" is the UK government standard form used when one is invited to seek alternate employment.

    Ahh, a "pink slip" then.

    Thanks for letting me (and others) know :)

  31. Fatman
    FAIL

    Re: New contractor

    <quote>Ahh, a "pink slip" then.</quote>

    Also known as walking papers.

  32. Alien8n Silver badge

    Re: New contractor

    Yup, basically he'd made so many mistakes at so many clients that he was fired for gross misconduct.

    The bit that beggared belief was that he was still expected to work out a notice period, which meant his last couple of weeks also involved watching him like a hawk. If he'd been working for me directly it would have been a straight forward "go home and don't come back until your disciplinary hearing" and then all his access removed there and then. Some people really don't understand how damaging a disgruntled employee can be. Reminds me of another conversation when engineering.

    "We're going to be making half the company redundant."

    "What are we doing to prevent sabotage of the equipment once we tell everyone?"

    "Nothing, no one would do that."

    Followed by half the factory going down over the next week as the operators started sabotaging the equipment.

  33. Christoph Silver badge

    Re: New contractor

    Which make it interesting when USA folks refer to the current (45th) president as 'P45' when they don't want to use his name.

  34. Antron Argaiv Silver badge
    Thumb Up

    Re: New contractor

    45*

    * indicating that he's "technically" the president, but there are extenuating circumstances...

  35. Nunyabiznes

    Re: New contractor

    ""We're going to be making half the company redundant."

    "What are we doing to prevent sabotage of the equipment once we tell everyone?"

    "Nothing, no one would do that."

    Followed by half the factory going down over the next week as the operators started sabotaging the equipment."

    Yep, been there, fixed that. Working at a brand new factory that replaced one in another state. All of the equipment we received from the old factory had to be refurbished because of sabotage. We figured it out pretty quick after the first *BANG*.

  36. Bill Gray

    Re: New contractor

    Hadn't heard of that person being referred to as P45. I occasionally refer to him as the Orange Lord, or as He-Who-Must-Not-Be-Named-While-Eating.

  37. DougS Silver badge

    45

    It has been common practice for at least the past few presidents to sometimes refer to them by their number. Not necessarily due to dislike - at least not STRONG dislike. People who truly dislike a president come up with far more colorful names.

  38. FishCounter

    Re: New contractor

    Sounded like an RGE to me; Resume Generating Event. You folks would term that a CVGE, I suppose.

  39. Mephistro Silver badge
    Happy

    In my first week in my first paid job...

    ... I was told by one of the execs to familiarize myself with a PC running THEOS/OASIS that was to be installed the next day at a customer's. I proceeded to test the software by inputting some fake data, generating queries and reports and all that carp.

    When I finished doing this, I decided to wipe all the data I created by using an "Initialize" option in the menu. To my indescribable horror, it wiped out all the data, the software and most of the OS.

    After twenty minutes of panic, cold sweats and frantic reading of the manuals, I found there was an option to undo the last initialization.

    Lesson learned: RTFM before playing with a system you aren't familiar with! 8^)

  40. Colin Bull 1
    Mushroom

    cp can also be dangerous

    After several years working in a DOS environment I got a job as project Manager / Sys admin on a Unix based customer site for a six month stint. On my second day I wanted to use a test system to learn the software more, so decided to copy the live order files to the test system.

    Unfortunately I forgot the trailing full stop as it was not needed in DOS - so the live order index file over wrote the live data file. And the company only took orders for next day delivery so it wiped all current orders.

    Luckily it printed a sales acknowledgement every time an order was placed so I escaped death and learned never miss the second parameter with cp command.

  41. Anonymous Coward
    Anonymous Coward

    not all heroes wear capes.

    or are heroes.

  42. Anonymous Coward
    Anonymous Coward

    it was like that when i logged.

  43. Anonymous Coward
    Anonymous Coward

    i'd written a script to deploy the latest changes to the live environment. worked great. except one day i'd entered a typo and it was now deploying the same files to the remote directory, over and again.

    it did that for 2 whole years with around 7 code releases. not a single person realised the production system was running the same code after each release with no change in functionality. all the customer cared about was 'was the site up?'

    not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years???

  44. Yet Another Anonymous coward Silver badge

    Look on the bright side, any bugs your team had introduced in those 2 years had been blocked by your intrinsically secure script

  45. Prst. V.Jeltz Silver badge
    Happy

    not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years???

    That is Classic! not surprised about the AC!

    Bet some of the beancounters were less than impressed , probly on customer side :)

  46. ma1010 Silver badge

    Sounds like a great idea!

    I'd written a script to deploy the latest changes to the live environment. worked great. except one day i'd entered a typo and it was now deploying the same files to the remote directory, over and again

    Could you PLEASE get a job sending out Windows 10 updates? Or almost any other updates, except for security ones?

  47. Anonymous Coward
    WTF?

    Time machine

    Can I have a go on your time machine? With the shiny modern macbook in a 1980s story!

  48. Robert Helpmann?? Silver badge
    Childcatcher

    Manual Recovery

    The man who one day would become my boss, let's call him "Dick"*, was called in with his fellow floor manager to discuss what needed to be done while their boss went on vacation. The way it was phrased was "When I get back, I want to look at our inventory and not see any of this stuff," motioning to the warehouse floor full of staged shipments. As soon as the boss was gone, Dick went into the inventory system and deleted everything. No inventory in the system. No record of where the stuff was supposed to be shipped. Gone! His counterpart on the next shift came in the next day and had to recreate everything from paper records. Unfortunately, rather than letting folks like that go from management positions, the company transferred them around so the pain never ends.

    * It actually was his name and I called him that every day.

  49. Antron Argaiv Silver badge

    DG/UX: I designed a low cost Token Ring card for those AViiON systems.

    Sadly, they came to a premature end when Motorola decided that there wasn't enough of a market for their 88000 processor, around which the systems were designed.

  50. The Mighty Spang

    way back on a Dec Vax

    operator was deleting a users account. now forgive me ive not touched vax syntax in a long time. IIRC he was trying to do

    del [...]*.*.*

    which should have deleted all the files in this folder and all folders below but did this

    del [*...]*.*.*

    which started at the root of the drive deleting everything under it... oops. back to the mag tapes.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018