"Two is one. And one is none".
Also: Schrödinger's backup.
The US Air Force Inspector General is investigating the corruption of around 100,000 investigation records, and presumably someone's asking hard questions about backups. Corruption in the database happened last month, but has only just come to light after both the USAF and Lockheed Martin threw in the towel on trying to …
"..and perhaps someone felt it looked bad?"
My thoughts exactly. From the article:
"The Automated Case Tracking System had data dating back to 2004 on complaints, investigations, appeals, and freedom of information requests, covering everything from waste and fraud to sexual harrassment."
Exactly the type of information many companies would be most happy to "accidently lose".
The Register speculates “aggressively leveraging” the vendor's capabilities conceivably covers things like yelling at various veeps, using lots of capital letters in e-mails, and sending copies of contracts off to lawyers.
You forgot big sticks with nails in it, and, if a BOFH is involved, "augmented" cattle prods.
Why is it that governments (and large corporations - not much difference in so many bad ways) are so terrible at specifying penalties for failure to perform? The real issue should be which Lockheed Martin exec is for the high jump, but I'm guessing that the contract will have enough weasel clauses that in the end either nobody will get penalized, or they'll find some low-level scapegoat to blame, probably the same person who wrote memo after memo warning "this system is broken".
Why is it that governments (and large corporations - not much difference in so many bad ways) are so terrible at specifying penalties for failure to perform
Because, sometimes in the 1990's the CEx's with Jack Welsh spearheading "the movement", figured out that the best way to rise above being a merely highly-salaried employee and make the transformation into the "Independently Wealthy"-class simply was to loot the shareholders.
"Performance" to these people is how large a fraction of profits they can nick before wasting them on the business or paying dividends.
Sometime penalty clauses don't help. If you get as far as the penalty clause you've already lost.
I might get 2 million euros because AWS lost my backups (for example) but now I haven't got a business anymore.
Much better to check that things work as expected and plan contingencies for things which matter to your business.... but then most large businesses don't really know what matters to them :-)
Yes, if you get the penalty clause, someone has definitely lost.
But it shouldn't be the one depending on the service. It should be one providing the service. They are, after all, supposedly the experts, which is why they got hired. If you lose a business because the data was lost (and yes, this does happen. All too often), the penalty should include the full cost of rebuilding that business. If a company isn't willing to agree to such penalty clauses, perhaps it's because they know they're fuckups in the making, aren't confident of their services, and thus shouldn't get the contract in the first place.
Yet we never see such penalty clauses, and we keep seeing large corporations and governments get fucked over again and again while the execs and their companies doing the deed never get penalized in a meaningful way.
"system admins saying better backup procedures were necessary."
That assumes there were backup procedures. There's no mention of those in any of the reports and presumably none in the original statement. That raises the question as to whether there were any backup procedures at all. What will be really interesting is if someone manages to restore data up to a particular point in time some years ago. Then the hunt starts for the memo that said they should stop wasting money on taking backups that never get used.
When I used to do recovery testing, it was a case of not only using backup media stored in another location but restoring to fresh hardware in an off-site location. If there is a fire you can't even assume the original hardware, physical network or even the building is available.
"restoring to fresh hardware in an off-site location"
And making sure the test system is wiped afterwards.
The first time you try this you'll probably learn a good deal about making backups. In my case it was discovering that /etc was stored a long way into the tape. We rearranged the file system backup sequence so that restoration gave us what we needed to start the database restoration quite early in the proceedings.
Interesting. a downvote. Presumably from someone who doesn't mind their backup going up in the conflagration or being stolen.
Chubb used to tell a story about their fire-safes. The Co-op in Belfast stored their backups in one of Chubb's safes in their HQ, same building as the computers. When the building burnt down the safe fell several floors and jammed shut. Rather than wait for the locksmith to arrive and sort it out someone decided to use a torch to burn their way into the safe (fire safes are designed to resist fires, not oxyacetylene torches). The backups next to the opening were destroyed by the torch.
I find that the instances of "accidents" goes up sharply when there's something embarrassing waiting in the wings to be processed, or when that is already in progress. I'm always suspicious that nobody really bothers to check that a decent backup routine was in place, and that it worked - not spotting that that takes a spectacular and frankly rather unbelievable absence of *anyone* with a clue in IT.
Of course, nothing will change. Maybe someone will have some awkward moments in front of the Senate, but that will be about the worst of it. At best they'll terminate a contractor who had nothing to do with it, but I don't expect anything more substantial to happen, and so ends whatever exposure had been in store for the person who orchestrated this "accident".
I know there's a limitless supply of stupidity in the world, but this one strains credulity.
Except Hillary is fucked because there was a backup and FBI has it (and God knows who else).
Even if Obama bites the bullet and pardons Hillary, the Clinton Foundation's prices for "services rendered" just tanked - now the price payable is to keep all that juicy data from "Evul Haxors and stuff". Not exactly a solid foundation for a lasting dynasty or retirement plan, IMO.
That's why everyone are so desperate over Trump - With all those skeletons in the closet, Hillary is now the perfectly controllable "everything-for-free" candidate, and "they" are losing that once-in-a-generation chance of having a true bargain-basement presidency bought at the very bottom of the market. To Trump. Who knows how to make deals. The Outrage! The Horror!! The Deprivation!!!
https://informedvote2016.wordpress.com/2016/03/18/do-i-really-need-to-worry-about-hillarys-emails-yes-she-will-be-indicted-full-form/
And the UK too. Still, since we sold the Americans all those noisy, expensive Harriers we need something to keep the bluewater canoe club in beer and chips (and that's real chips, as in 1/2" thick chunks of potato fried in oil & eaten with large chunks of cod or haddock and not those skinny, limp bits of salt-encrusted white fluff sold as "French fries" or the flat slices that we call crisps).
The whole of "the Automated Case Tracking System [which] had data dating back to 2004 on complaints, investigations, appeals, and freedom of information requests, covering everything from waste and fraud to sexual harassment" was on one single disk/tape/floppy/punchcard?
All of it?
So presumably at every point during the last 12 years that someone in Lockheed Martin said:
"Hey, why not have a backup? Or RAID? Or printout?" they ran off to check with the in-house lawyers who replied:
"Nah. Nothing in the contract about that. Just leave it as it is..."
Lockheed Martin notified the Air Force after it spent two weeks trying to recover the information.
You can just imagine how much perspiration dripped into that keyboard...
Lockheed understand that all federal systems are accountable to FISMA/NIST.
There was a detailed recovery plan that was certified annually by a small team of experts.
Someone had to certify annually that recovery tests had been performed and validated.
Lockheed probably billed for the time to update and test recovery plans. No bonuses for this one.
"Oh, NO! We lost the database with the details of all the tortures we did. What a shame."
"Oh, NO! We lost the database with the details of all the complaints folks have made about us. What a shame."
"Oh, NO! We lost the database with the details of [fill in whatever you'd like to see "missing"]...
"Oh, NO! We lost the database with the details of [fill in whatever you'd like to see "missing"]...
There's a long history of this sort of problem in the US Department of Defense. Protecting vast stacks of paper in the National Personnel Records Center in an era when many employees smoked was obviously given the foremost attention:
"On each of the floors were large spaces for records storage, stretching hundreds of feet and containing no firewalls or other firestopping to limit the spread of fire. ... The entire facility lacked heat or smoke detectors to automatically detect fire or a fire sprinkler system to automatically extinguish fire."