back to article You shrunk the database into a .gz and the app won't work? Sigh

Welcome yet again to On-Call, our unimaginatively-named regular recap of readers' recollections of jobs gone wrong. You all liked a double-barrelled story so much last week we've decided to do it again, starting with “Gary” wrote to tell us of his time working for Burroughs and the job he was sent to do for a bank with a …

Page:

  1. Tom 7

    Replace tape

    When trying to restore a backup for someone we discovered the tape was the 2nd part of the backup.

    Where's the first tape?

    That's all there is.

    When you run the backup does it ask for the tape to be replaced?

    Yes, we pop it out and pop it back in again and after a bit it comes up with 'finished successfully'.

    Turns out this has been going on for ages and its only the customers close resemblance to Susanna Hoffs that encourages an almost certainly futile attempt to recover individual bits of disk and compare them with the corrupted database and the eventual disappointment of actually being able to stitch the database successfully back together, rewriting the backup to check that fresh tapes are being inserted, running a backup and verifying and then wending off into the night only to be punched by my trainee in harness for not spending time pretending we weren't there yet even thought running and verifying the backup twice 'just to be sure' had got the customer looking sideways in a definitely not alluring way.

    1. Anonymous Coward
      Anonymous Coward

      Re: Replace tape

      first job as sysadmin-in-training, couple of weeks in, now going to be responsible for changing backup tapes .... person who did it before showed me the pop the tape out, pop it in again, backup software said "finished successfully" routine.

      Me: "Is it always that quick?"

      Supervisor: "yes, not much changes on this server during normal days"

      Me, later, looking at tape to put in for next night, asking self - 'why does it say 'cleaning tape' on it?'

      1. Guy Geens

        Re: Replace tape

        This reminds me:

        Technician is called in to fix a tape drive. After he's done, the operator pops in a tape to test if everything is OK. After a few seconds, the tape comes out again. The operator says: "Everything is OK, backup is done"

        The technician is highly suspicious and takes a look at the console:

        INITIALIZING TAPE...

        SYNTAX ERROR

        After a simple edit to the script, the backup takes considerable longer. The operator looks at a cupboard filled with tapes and realizes these are all empty...

        1. Alan Brown Silver badge

          Re: Replace tape

          The good thing about packages like Bacula is that they actually READ the tape before writing to ensure people haven't pulled that kind of stunt.

          (They also use a database which contains SHA256 and exact location of every file on every tape so restoring XYZ file doesn't mean having to run hours of restores, AND you can tell when the file last changed.)

      2. Chris King

        Re: Replace tape

        You probably had one of the cleanest drives out there, I inherited a really dirty one.

        I knew it was going to be trouble when the report from the DEC engineer said (and it was all-caps)

        "THIS DRIVE CONTAINED SO MUCH DIRT, YOU COULD PROBABLY GROW POTATOES IN IT !!!"

        1. Anonymous Coward
          Anonymous Coward

          Re: Replace tape

          Clearly not the same DEC engineer who let the ash on his cigarette (yes, really, in the computer room) grow so long that it all dropped into the tape reader/punch on the PDP15/76.

        2. Alan Brown Silver badge

          Re: Replace tape

          "THIS DRIVE CONTAINED SO MUCH DIRT"

          I just pulled a DLT drive off the bottom shelf in a room which meets this description.

          The room itself was fairly clean. People forget that drives are in cases with fans and that tends to suck every bit of airborne dust available into the case.

          Rather fine dust, finer than talc, over a 15 year period....

    2. Anonymous Coward
      Anonymous Coward

      Re: Replace tape

      Please tell me that happened on a Monday.

      1. Anonymous Coward
        Anonymous Coward

        Re: Replace tape

        The cleaning tape incident? I have to admit it was about 20yrs ago, and I don't remember day of week, but been thinking about it today and I vaguely remember taking on the task half-way through a month - the supervisor's boss had insisted on the changeover mid-month to avoid any risk to month end data.

    3. Zippy's Sausage Factory

      Re: Replace tape

      I finished reading this and got three comments further down until I realised I wasn't paying any attention and just thinking about Susanna Hoffs.

      Saddo that I am, I was actually thinking "I know that name from somewhere, who is she?"

      Ah yes, Bangles. My one chance to see them we got lost - wandering around Hyde Park with "Walk Like An Egyptian" blaring out...

      1. Locky
        Flame

        Re: Replace tape

        @Zippy's Sausage Factory - Famously told by an unscrupulous producer that Eternal Flame would sound better if she recorded naked

        Now try and get that out of your head....

      2. usbac Silver badge

        Re: Replace tape

        @ Zippy's Sausage Factory

        This is a little bit off-topic, but still relates to tapes...

        Way back in the mid 80's I was in school in Los Angeles to be a recording engineer. Our school had a very high end recording studio. Some of us that had earned the trust of our professor were allowed access to the studio on off hours. We had friends with bands that wanted to record demos to send in to record companies (those were the days).

        The problem was that reels of 2-inch tape were very expensive for us poor students. A reel of good quality tape could be as much as $200-300. Our professor told us where to get really good tape for cheap. He explained that the high-end recording studios sell reels of "used" tape for really cheap. The reason is that when well known bands come in to record an album, they run many takes of the same song. Since they are paying thousands of dollars per hour to use the studio, they just keep grabbing new reels of tape. When the project is over, the studio bulk erases the tape, and sells them off cheap.

        A friend and fellow student had some friends of his that wanted to record a few demos. So, he went and bought a few reels of this used tape. We arrange to use the studio ofter class one afternoon. His friends aren't quite there yet, so we start getting things ready in the control room. I take one of the reels and start loading it up on our 24-track deck. I notice that a bunch of the VU meters twitch as I'm winding the tape onto the take-up reel. We both look at each other, and realize this tape wasn't erased. So, we quickly re-patch the console for playback instead of recording.

        We start listening to the tracks, and make a rough mix. As we are listening we're thinking we've never heard this song before, and are trying to figure out who is the band. About this time Richard's friends show up. They come into the control room and say "Cool.. The Bangles... where did you get this?" It turns out we had an un-released song.

        There was one mostly usable take on the tape. We quickly unloaded the tape and put it away. I don't know what Richard ever did with the tape. These days, we could have done a decent mix-down, and posted it on the net. We were both too honest do anything unscrupulous with the tape. It would be quite an artifact to have these days, however.

        1. JPeasmould

          Re: Replace tape

          When I worked at Wessex Studios in London in the eighties we cleared out our tape store once a year. We didn't wipe the tape and resell it as it had already been paid for.

          The first year I just took the sides off the reels and let them unwind into bin bags. I spent the next day picking the tape out of the trees all along Highbury New Park (a good half mile) due to the antics of a couple of the local intelligentsia.

          The second year I used a splicing blade to cut the tape from the reels. You can't get as much fun from 6" lengths of tape.

          I did find all the "Never Mind the Bollocks" masters in the attic, but they had a bit of masking tape round them stating that they were being kept for the court (the Sex Pistols were fighting Virgin at the time). I assume Matrix got them as they were not claimed by the time Chrysalis sold Wessex.

          I wish I'd nicked them along with the Reg Dwight demos...

      3. William 3 Bronze badge

        Re: Replace tape

        With regards to the Susanna comments, surely only doing work because of how that woman looked was nothing more than everyday sexism, and given how left leaning and all SJW all the comments and articles are in the register these days I can only wonder why it was voted up so highly.

        Perhaps because all the people in this section of the register are actually skilled workers rather than snot nosed left wing SJW humanities degrees tossers that go looking for offence as justification for their existence that populate the other areas of the register.

        1. Anonymous Coward
          Anonymous Coward

          Re: Replace tape

          ... given how left leaning and all SJW all the comments and articles are in the register these days ...

          You must be reading a version of TheRegister from some other, bizarre dimension.

          1. 404

            Re: Replace tape

            OP has a valid point - mention anything negative about Obama, Hillary, <insert progressive sacred cow here>, and the downvotes multiply like rabbits. Not Reddit-level SWJism, but still present in different forms.

            1. John Gamble

              Re: Replace tape

              Or, perhaps because the negative comments on the alleged sacred cow are about as dull as the "will it play Crysis?" comments?

              Here's a clue: if the commentard actually uses the term "SJW", you can safely assume too many drugs were consumed during the commentard's fetal development.

              1. Anonymous Coward
                Anonymous Coward

                Re: Replace tape

                But we need to know, does Obama play Crysis?

            2. kain preacher

              Re: Replace tape

              Could it be that most of the negatives I've seen about Obama and hillary are from a few rabid posters her that post made up stuff ? I mean if you want to slag off on Hillary for using a personal email serve by all means do it, but dont act like she was the first or because all dems are corrupt. Just remeber Collin powel used AOL.

              I for one wished that Obama had not chicken out and did not make weed legal or pardon Snowden. I would also like for him to say that Julian will not be prosecuted. If nothing more than to then deflate Julian's ego.

    4. Stoneshop

      Re: Replace tape

      Back in the days of VAX11/780's and such, one of my colleagues was called to fix a disk drive. THE disk drive, OS, data, everything. And so he does, then proceeds to help with restoring the backup. Which is duly retrieved from the safe, but he's rather surprised to see that it's just three RX01 floppies. Which is rather at odds with the capacity of the disk, 120MB, where an RX01 can only hold a few 100kB.

      "So, how do you run your backup?"

      "Well, like the manual says, @SYS$SYSTEM:STABACKIT, and we swap floppies when prompted"

      Every Friday the customer had faithfully made a bootable floppy set for the VAX to run off, so that the system disk was not in use when you wanted to backup or restore it. No actual backup, however...

    5. ecofeco Silver badge

      Re: Replace tape

      Back up tapes at one place I worked were somehow not working after I began testing them. One after the other. Ran verification tests and still nothing or only partial, corrupted back ups.

      Spent weeks trying to figure it out, then by pure happenstance I notice a very strong magnetic field in the vicinity of where the tapes were stored. (strange buzzing noise from a piece of equipment that should not be making buzzing noise but only when sitting in a very certain spot)

      Managed to get a building engineer to come down with a field tester and sure enough, there was an unusually strong magnetic field right in that one spot. Turns out it was close to an outside wall that was receiving very strong bounced radio waves from somewhere outside.

      You can surmise the rest. Yeah, no backups.

      1. BostonEddie

        Re: Replace tape

        recollects me of the time the local (Mass) office of the IRS moved and consolidated their new operation center into one location. There, right in the middle of the new location, was a high voltage line feed power pole exposed for all to see in the middle of the storage area. Most unsightly. Why don't we hide that ugly thing by putting those reels of magnetic tape around it...

        Income taxes were a bit late that year.

    6. This post has been deleted by its author

  2. Anonymous South African Coward Bronze badge

    Backup tapes...

    Back yonkers one of our clients did his usual backup, and locked the tapes up in his fireproof safe (onsite). This was in the heady days of Novell 3.12...

    Came back the next day, PC's was gone. Server was gone. Safe (with tapes inside) was gone. No hard copies.

    Oops.

    And the taxman was busy with an audit on him...

    1. allthecoolshortnamesweretaken

      Re: Backup tapes...

      "PC's was gone. Server was gone. Safe (with tapes inside) was gone. No hard copies. [..] And the taxman was busy with an audit on him..."

      What an odd coincidence.

      1. JimC

        Re: Backup tapes...

        Why would you expect a serious robbery *not* to take a safe unless its so large as to be impractical to move? There are enough of us here and with long enough careers that most unlikely events are going to turn up...

        And theft of all IT equipment including all the tapes is very familiar. And on at least one occasion it was followed by "Oh, is that why you told us we must always get someone to take a tape home? We decided it was too much trouble." (as if we hadn't explained why they needed off site).

        1. Chris King
          Facepalm

          Re: Backup tapes...

          An academic decided that he was going to teach "IT security" to his final year Informatics students, stressing the importance of protecting physical kit and taking regular backups.

          Next day, he plonks his laptop on a desk, turns his back, and someone runs off with it.

          Bonus points for losing three weeks of lesson plans, presentations and course materials that weren't backed up anywhere else.

        2. DropBear

          Re: Backup tapes...

          "Why would you expect a serious robbery *not* to take a safe unless its so large as to be impractical to move?"

          Because if it isn't bolted to the floor* it's called "security theatre" not "a safe", unless you need a forklift to move it (and even then, you should know better).

          *if said safe has its own floor, complete with giant-ass submarine-style hatch door weighting many tons, you officially have my permission to not bolt it to anything and still call it a safe. Although you might prefer to call it "a vault"...

          1. I ain't Spartacus Gold badge
            Happy

            Re: Backup tapes...

            I had a safe by my desk in a previous job. It used to be the finance manager's office - but he'd changed. Turns out the reason is that the safe door broke, so rather than remove the safe, he simply ordered a new one to be delieved to a different office, and swapped...

        3. kain preacher

          Re: Backup tapes...

          Jim I would follow you thought process except for it was right before an audit.

  3. GrumpenKraut
    Facepalm

    Editing a config file with an Office program

    ... similar "success". Not me, but the "IT specialist" of a not really small company.

    Had the guy read the error messages over the phone. The XML-ish tags gave it away. He wasn't able to copy the old version of the file, that is, using the 'cp' command. I had to come to the company. Charged him 3 hours and promised not to tell anyone.

    1. Anonymous Coward
      Anonymous Coward

      Re: Editing a config file with an Office program

      I had a job setting up some data ETL into a new and pricey proprietary system from a legacy database using a very well known and established XML standard. A simple task that ended up being far more difficult that expected. After much tinkering, transformation and experimentation I eventually used some test data from the proprietary tool to generate some of it's *own* XML which I copied to my Unix box from which the system duly refused to import as well. Aha, with my diligent troubleshooting notes and sample files in hand, I emailed the vendor's tech support email address and got a phone call the next day, from a developer no less.

      Vendor: "We've figured out your issue"

      Me: "Ah Great! What is it???"

      Vendor: "Your XML file is missing ctril-m characters from the end of each line."

      Me: "Ummmm....ctrl-m, you mean DOS line end of line characters? They are outside the tags anyway."

      Vendor: "The what?"

      Me: "Ummmm... the tags..." (explanation of XML and tags follows)

      Vendor: "Those angle bracket thingies? Why would I try to parse those when it is so much easier to parse line by line?" (I paraphrase, but only just a bit)

      Me: "........"

      Things got somewhat interesting after that as my new boss had recommended buying this thing and had a strong motivation to "Make It Work", I'll not bore you with the predictable details except astoundingly, the next tier of management agreed with my findings when it all came to a head.

      1. Doctor Syntax Silver badge

        Re: Editing a config file with an Office program

        Sub-contracting to a sub-contractor for the main contractor (Your first guess is probably right) public sector job. We were to take XML encoded documents from main contractor for processing. Very early in the proceedings I had to take a trip to visit the onshore representative of the main contractor's favoured development house to explain how to handle apostrophes and the like in XML. Not surprising that from time to time we had outbreaks of badly formed documents sent to us.

      2. Kiwi
        Joke

        Re: Editing a config file with an Office program

        the next tier of management agreed with my findings when it all came to a head.

        I believed you all the way up until that statement! :)

    2. JoeF

      Re: Editing a config file with an Office program

      Had somebody edit a Linux config file with a DOS/Windows text editor which put crlf instead of lf in there.

      Was rather hard to detect because vim (vi improved, cough) "helpfully" hides control characters, so the file looked ok.

      A hexdump finally revealed the problem.

      1. Justin Clift

        Re: Editing a config file with an Office program

        From rough memory, vim has a "binary" mode which can show them as well.

        I think the -b switch from the command line turns it on.

        eg:

        $ vim -b some_file.txt

        But yeah, hex editing utils also work. :)

    3. Anonymous Coward
      Anonymous Coward

      Re: Editing a config file with an Office program

      I've had a nice one involving UTF-8 apostrophes in usernames. Lovely Irish girl had been resetting her password every day for months in order to log in.

  4. dvd

    Went to Otley instead of Otley road?

    Pah! I worked with someone who went to the wrong African COUNTRY on a support call.

    In fairness to him it was the travel agents fault, not his. He ended up getting a taxi to the border and walking across in a Checkpoint Charlie sort of deal.

    1. Andrew Moore

      We had an American colleague fly to Iceland instead of Ireland. Only realised it when he got into a taxi at Reykjavik and asked to be taken to Dublin city centre.

      1. Fibbles

        We had an American colleague fly to Iceland instead of Ireland. Only realised it when he got into a taxi at Reykjavik and asked to be taken to Dublin city centre.

        Ok, just a second whilst I get the pontoons and outboard motor from the boot.

      2. Phil W

        "We had an American colleague fly to Iceland instead of Ireland. Only realised it when he got into a taxi at Reykjavik and asked to be taken to Dublin city centre."

        If he didn't notice until he was in the taxi and asked to go to Dublin city centre, was he phenomenally ignorant/stupid? To get that far surely he musn't have noticed:

        - The fact it said Reykjavik rather than Dublin on his tickets

        - The various signs in Icelandic (I suppose he could have thought it was Gaelic, but come on)

        - The length of flight (you didn't say where he was flying from, presumably the USA. But we're talking about an extra hour or two or flight at least most likely, I would of thought it would be noticeable)

        - The Captain announcing the destination as Reykjavik on take off, and before approach.

        - How bloody cold it was! Either in the airport or when he went out to get the taxi. I know Ireland is not exactly tropical but the difference is usually fairly substantial I believe? I think Reykjavik peaks at about 13 degrees in Summer?

        1. Doctor Syntax Silver badge

          "If he didn't notice until he was in the taxi and asked to go to Dublin city centre, was he phenomenally ignorant/stupid?"

          He was American. He'd flown over water for some time. He was somewhere furrin'. It's all the same, isn't it?

          1. TomPhan

            Flying into Reykjavik and the only clue was when in a taxi? Wouldn't it have happened after he landed at Keflavik?

          2. Anonymous Coward
            Anonymous Coward

            Having recently flown to America via Iceland I really have no idea how one could get it confused.

        2. This post has been deleted by its author

    2. rh587

      Pah! I worked with someone who went to the wrong African COUNTRY on a support call.

      Pah. Just him, not anyone else?

      I know someone who sent a shipment of 300 new cars to Sweden instead of a Swindon dealer (large dealer, it was the order for the new plates, huge turnover in a couple of weeks).

      I wish I were joking... it shouldn't even be possible, but he managed to bork the paperwork sufficiently that the error wasn't uncovered until his Swedish importer called asking why the port was pestering him over 300 cars on their quayside with his name on that he hadn't ordered...

    3. John Brown (no body) Silver badge

      "Pah! I worked with someone who went to the wrong African COUNTRY on a support call."

      Bollocks! You beat me! We had our south west engineer go to a call in Poole, Dorset, pre-sat-nav and smartphone era. He rang back after a 2 hour drive saying he can't find the street. He'd even asked around and none of the locals knew it either. Bloody call centre had got confused and should have sent one of the northern engineers to Poole, near Otley in North Yorks.

    4. kain preacher

      I got one better for you. I was traveling on this road. lets Call it 1st. Now first street has the destination of north on one side and south on the other side. Now when you cross over to the next city it's east and west, and no there is no sign letting you know you changed cities.

  5. Anonymous Coward
    Anonymous Coward

    When life imitates art...

    One of my favourite quotes on bash.org was a guy who claimed he'd managed to lose a PC inside his apartment. It responded to ping, he just couldn't work out where it was.

    Last job was as part of a network team looking after a nationwide estate for an outsourcer. Got dragged into an emergency call with management where a server had gone missing on a campus site and they wanted my help tracking it down. True to art, the server (an ancient Windows box) responded to ping, RDP and SMB, but it was missing from the rack and noone could work out where it had gone.

    An hours discussion with the team who managed the other half of the campus network on this site (a horrible legacy arrangement) revealed it was still in the same building and still in the same room, and that the network ports assigned to it hadn't been changed.

    The server was eventually located, in the rack it was meant to be. Turns out the sticker for the rack had fallen off, and the rack next to it wasn't stickered, so some well-meaning newbie had come in and taped the sticker they found on the floor onto the wrong rack, then couldn't work out why they couldn't find it when they were asked to look for it a few weeks later...

    1. Anonymous Coward
      Anonymous Coward

      Re: When life imitates art...

      You Could have told them which switch and port to go follow the cable from.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like