back to article 12 years of US Air Force complaints lost in database crash

The US Air Force Inspector General is investigating the corruption of around 100,000 investigation records, and presumably someone's asking hard questions about backups. Corruption in the database happened last month, but has only just come to light after both the USAF and Lockheed Martin threw in the towel on trying to …

  1. RIBrsiq
    Alert

    "Two is one. And one is none".

    Also: Schrödinger's backup.

  2. lsces

    So what software was being used?

    The other question that comes to mind is when using a 'distributed system' who is responsible for backing up?

    1. Anonymous Coward
      Anonymous Coward

      Re: So what software was being used?

      CommVault

  3. Aaiieeee

    Too many complaints

    ..and perhaps someone felt it looked bad?

    1. Mark 85
      Devil

      Re: Too many complaints

      Or perhaps this database also had the problems and complaints about the F35?

    2. John Tserkezis

      Re: Too many complaints

      "..and perhaps someone felt it looked bad?"

      My thoughts exactly. From the article:

      "The Automated Case Tracking System had data dating back to 2004 on complaints, investigations, appeals, and freedom of information requests, covering everything from waste and fraud to sexual harrassment."

      Exactly the type of information many companies would be most happy to "accidently lose".

  4. wolfetone Silver badge
    Coat

    Must've been stored on 8 inch floppy disks.

  5. Potemkine Silver badge

    Well done BOFH

    That's a very efficient way to close all the tickets in one shot.

    1. Jim 43

      Re: Well done BOFH

      This is a very efficient way to condense all the existing tickets to one ticket.

  6. Michael H.F. Wilkinson Silver badge
    Joke

    The Register speculates “aggressively leveraging” the vendor's capabilities conceivably covers things like yelling at various veeps, using lots of capital letters in e-mails, and sending copies of contracts off to lawyers.

    You forgot big sticks with nails in it, and, if a BOFH is involved, "augmented" cattle prods.

    1. SolidSquid

      If a BOFH is involved they're probably the ones who deleted the complaints. Oh sorry, "tested" the backups of the database

      1. Fungus Bob

        If a BOFH is involved, there's no way to tell.

        1. John Tserkezis

          Correction:

          If a good BOFH is involved, there's no way to tell.

          1. Fungus Bob

            No such thing as a lousy BOFH.

    2. Anonymous Coward
      Anonymous Coward

      translate from 'merkin

      What we'd call a tyre iron, they'd call a tire-lever - I picture aggressive leveraging to involve a metre long piece of bent steel

  7. raving angry loony

    Bad contracts

    Why is it that governments (and large corporations - not much difference in so many bad ways) are so terrible at specifying penalties for failure to perform? The real issue should be which Lockheed Martin exec is for the high jump, but I'm guessing that the contract will have enough weasel clauses that in the end either nobody will get penalized, or they'll find some low-level scapegoat to blame, probably the same person who wrote memo after memo warning "this system is broken".

    1. fajensen

      Re: Bad contracts

      Why is it that governments (and large corporations - not much difference in so many bad ways) are so terrible at specifying penalties for failure to perform

      Because, sometimes in the 1990's the CEx's with Jack Welsh spearheading "the movement", figured out that the best way to rise above being a merely highly-salaried employee and make the transformation into the "Independently Wealthy"-class simply was to loot the shareholders.

      "Performance" to these people is how large a fraction of profits they can nick before wasting them on the business or paying dividends.

      1. Anonymous Coward
        Anonymous Coward

        Re: Bad contracts

        Sometime penalty clauses don't help. If you get as far as the penalty clause you've already lost.

        I might get 2 million euros because AWS lost my backups (for example) but now I haven't got a business anymore.

        Much better to check that things work as expected and plan contingencies for things which matter to your business.... but then most large businesses don't really know what matters to them :-)

        1. raving angry loony

          Re: Bad contracts

          Yes, if you get the penalty clause, someone has definitely lost.

          But it shouldn't be the one depending on the service. It should be one providing the service. They are, after all, supposedly the experts, which is why they got hired. If you lose a business because the data was lost (and yes, this does happen. All too often), the penalty should include the full cost of rebuilding that business. If a company isn't willing to agree to such penalty clauses, perhaps it's because they know they're fuckups in the making, aren't confident of their services, and thus shouldn't get the contract in the first place.

          Yet we never see such penalty clauses, and we keep seeing large corporations and governments get fucked over again and again while the execs and their companies doing the deed never get penalized in a meaningful way.

    2. Mark 85

      Re: Bad contracts

      Quite possibly it's a variation of the "good ol' boy" syndrome. If you use the performance penalty clause on company "A", it will probably be used against you. I suppose there is a bit of honor amongst thieves in this case.

    3. John Brown (no body) Silver badge

      Re: Bad contracts

      "they'll find some low-level scapegoat to blame,"

      The PHB or the PFY. I'm torn. Could be either. Or both.

  8. Andy Non Silver badge
    Coat

    They probably lost all the records from the

    system admins saying better backup procedures were necessary.

    1. Doctor Syntax Silver badge

      Re: They probably lost all the records from the

      "system admins saying better backup procedures were necessary."

      That assumes there were backup procedures. There's no mention of those in any of the reports and presumably none in the original statement. That raises the question as to whether there were any backup procedures at all. What will be really interesting is if someone manages to restore data up to a particular point in time some years ago. Then the hunt starts for the memo that said they should stop wasting money on taking backups that never get used.

      1. Jim 43

        Re: They probably lost all the records from the

        Every Federal agency is required to comply with NIST 800 series guidelines within a year of publication. NIST 800-34 is the relevant document here.

  9. fajensen
    Angel

    Well, that is *one* way of meeting KPI targets and kick off those performance related bonuses. Just in time for Christmas too.

  10. David Roberts
    WTF?

    A backup isn't a backup

    Until it has been successfully restored?

    Or does this involve too much hard work?

    1. Doctor Syntax Silver badge

      Re: A backup isn't a backup

      Also, a backup isn't a backup until it's removed from the system and securely stored at another location.

      1. Andy Non Silver badge

        Re: A backup isn't a backup

        When I used to do recovery testing, it was a case of not only using backup media stored in another location but restoring to fresh hardware in an off-site location. If there is a fire you can't even assume the original hardware, physical network or even the building is available.

        1. Doctor Syntax Silver badge

          Re: A backup isn't a backup

          "restoring to fresh hardware in an off-site location"

          And making sure the test system is wiped afterwards.

          The first time you try this you'll probably learn a good deal about making backups. In my case it was discovering that /etc was stored a long way into the tape. We rearranged the file system backup sequence so that restoration gave us what we needed to start the database restoration quite early in the proceedings.

          1. Captain Scarlet Silver badge

            Re: A backup isn't a backup

            Restoring to fresh hardware, I remember the headaches caused by testing and finding out on several tests the system didn't actually have the capacity to restore to it (Was a PFY then).

        2. Linker3000

          Re: A backup isn't a backup

          Also never assume that the people who made and managed the backups are still around to restore them. Documentation - stored in several places FTW.

      2. Doctor Syntax Silver badge

        Re: A backup isn't a backup

        Interesting. a downvote. Presumably from someone who doesn't mind their backup going up in the conflagration or being stolen.

        Chubb used to tell a story about their fire-safes. The Co-op in Belfast stored their backups in one of Chubb's safes in their HQ, same building as the computers. When the building burnt down the safe fell several floors and jammed shut. Rather than wait for the locksmith to arrive and sort it out someone decided to use a torch to burn their way into the safe (fire safes are designed to resist fires, not oxyacetylene torches). The backups next to the opening were destroyed by the torch.

  11. Anonymous Coward
    Anonymous Coward

    Was this really an accident?

    I find that the instances of "accidents" goes up sharply when there's something embarrassing waiting in the wings to be processed, or when that is already in progress. I'm always suspicious that nobody really bothers to check that a decent backup routine was in place, and that it worked - not spotting that that takes a spectacular and frankly rather unbelievable absence of *anyone* with a clue in IT.

    Of course, nothing will change. Maybe someone will have some awkward moments in front of the Senate, but that will be about the worst of it. At best they'll terminate a contractor who had nothing to do with it, but I don't expect anything more substantial to happen, and so ends whatever exposure had been in store for the person who orchestrated this "accident".

    I know there's a limitless supply of stupidity in the world, but this one strains credulity.

    1. PNGuinn
      FAIL

      Re: Of course, nothing will change @ ac

      Rubbish.

      Lessons will be learned ...

    2. Zippy's Sausage Factory

      Re: Was this really an accident?

      Watch and see who gets big fat bonuses and/or cushy new assignments. If they post the previous manager to the coldest, harshest, worst-paid posting in Alaska, that'll also be an indicator...

    3. SoloSK71
      Megaphone

      Re: Was this (not) really an accident?

      just in time for some of the major inquiries into why the air force is crap at a) preventing and b) dealing with sexual harassment and abuse ... what a shock

  12. Anonymous Coward
    Anonymous Coward

    "...what difference does it make?"

    If no back ups are good enough for Hillary and the IRS, then no back ups should be perfectly fine for the USAF.

    1. Anonymous Coward
      Anonymous Coward

      Re: "...what difference does it make?"

      Except Hillary is fucked because there was a backup and FBI has it (and God knows who else).

      Even if Obama bites the bullet and pardons Hillary, the Clinton Foundation's prices for "services rendered" just tanked - now the price payable is to keep all that juicy data from "Evul Haxors and stuff". Not exactly a solid foundation for a lasting dynasty or retirement plan, IMO.

      That's why everyone are so desperate over Trump - With all those skeletons in the closet, Hillary is now the perfectly controllable "everything-for-free" candidate, and "they" are losing that once-in-a-generation chance of having a true bargain-basement presidency bought at the very bottom of the market. To Trump. Who knows how to make deals. The Outrage! The Horror!! The Deprivation!!!

      https://informedvote2016.wordpress.com/2016/03/18/do-i-really-need-to-worry-about-hillarys-emails-yes-she-will-be-indicted-full-form/

      1. SoloSK71

        Re: "...what difference does it make?"

        here's your sign

        and your tinfoil hat

  13. Anonymous Coward
    Anonymous Coward

    DBCC CHECKDB

    Because if you're backing up corrupted data, then what's the point?

  14. JJKing

    This would be the F-35 backup since it really doesn't work either. Now if only Australia was as smart as the Canadians and withdrew from this steaming pile of crap.

    1. Anonymous Coward
      Anonymous Coward

      @ JJKIng

      And the UK too. Still, since we sold the Americans all those noisy, expensive Harriers we need something to keep the bluewater canoe club in beer and chips (and that's real chips, as in 1/2" thick chunks of potato fried in oil & eaten with large chunks of cod or haddock and not those skinny, limp bits of salt-encrusted white fluff sold as "French fries" or the flat slices that we call crisps).

  15. Anonymous Coward
    Anonymous Coward

    Let the sueballs begin ...

    I wonder what the legal attitude to legal claims is in the US.

    Presumably, the lack of these records means that the USAF has no rebuttal evidence to the most outlandish of claims ....

  16. SW10
    WTF?

    Wait..

    The whole of "the Automated Case Tracking System [which] had data dating back to 2004 on complaints, investigations, appeals, and freedom of information requests, covering everything from waste and fraud to sexual harassment" was on one single disk/tape/floppy/punchcard?

    All of it?

    So presumably at every point during the last 12 years that someone in Lockheed Martin said:

    "Hey, why not have a backup? Or RAID? Or printout?" they ran off to check with the in-house lawyers who replied:

    "Nah. Nothing in the contract about that. Just leave it as it is..."

    Lockheed Martin notified the Air Force after it spent two weeks trying to recover the information.

    You can just imagine how much perspiration dripped into that keyboard...

    1. Jim 43

      Re: Wait..

      Lockheed understand that all federal systems are accountable to FISMA/NIST.

      There was a detailed recovery plan that was certified annually by a small team of experts.

      Someone had to certify annually that recovery tests had been performed and validated.

      Lockheed probably billed for the time to update and test recovery plans. No bonuses for this one.

    2. Linker3000

      Re: Wait..

      > > > Hey, why not have a backup? >>>Or RAID?<<<

      You really didn't write that, did you!? Hand in your sysadmin card on the way out. Kthxbye.

  17. Dave 32
    Pint

    Edward Snowden

    Has anyone thought to check with Edward Snowden to see if he has copies?

    Dave

  18. ma1010
    Black Helicopters

    Oops, we did it again!

    "Oh, NO! We lost the database with the details of all the tortures we did. What a shame."

    "Oh, NO! We lost the database with the details of all the complaints folks have made about us. What a shame."

    "Oh, NO! We lost the database with the details of [fill in whatever you'd like to see "missing"]...

    1. cray74

      Re: Oops, we did it again!

      "Oh, NO! We lost the database with the details of [fill in whatever you'd like to see "missing"]...

      There's a long history of this sort of problem in the US Department of Defense. Protecting vast stacks of paper in the National Personnel Records Center in an era when many employees smoked was obviously given the foremost attention:

      "On each of the floors were large spaces for records storage, stretching hundreds of feet and containing no firewalls or other firestopping to limit the spread of fire. ... The entire facility lacked heat or smoke detectors to automatically detect fire or a fire sprinkler system to automatically extinguish fire."

  19. Fatman
    Coat

    I KNOW HOW this happened......

    The root cause was quite simple....

    some dumbass emptied the Trash/Recycling bin.

    </snark>

    1. allthecoolshortnamesweretaken

      Re: I KNOW HOW this happened......

      Or maybe "Little Bobby Tables"...

      Anyway, just phone PLA Unit 61398 (61398部队) and politely ask for their backup.

  20. cray74
    Headmaster

    For the IT impaired...

    Is there a root cause analysis of why you couldn't recover this database from an offsite backup?

    Or at least a general theory of why this is FUBAR'd?

  21. Anonymous Coward
    Anonymous Coward

    don't worry

    I'm sure someone has a private email server with secure backups...

  22. Shovel

    They had to protect Hillary again

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon