Guess they are secure now.
It was with "great regret" that Orbit, makers of an app for professional childcare services, informed its customers that it lost all of their data during a weekend site upgrade – before discovering their backups hadn't been working for a year. Orbit describes itself as "a secure website" which provides "the Orbit Early Years …
The *ONLY* perspective from which Cloud makes sense is from the company that provides it. For users, it's a stupid idea, always.
There is *no* reason whatsoever for this app to have been storing data anywhere except locally with the users. That would have given superior security and reliability.
Sure, they could have replicated a small subset of that data back to their server if (and only if) it is needed for collaboration features, but to put *everything* in the cloud, that's where most apps fall over, and it's driven by greed - of the cloud providers - to be able to mine all the data and sell it to advertisers/statisticians/crooks as they please.
The most ridiculous part of all is that they were storing the only copy of the data on one server, which is 'doing it wrong' even from a cloud perspective.
What person smart enough to make backups in the first place fails to check to make sure the backups are useable? You make a backup & then immediately verify that the backup can be restored to recover the data.
A company charging money for services should be doing this constantly to a test server from the main server's backups, in order to make DAMN sure the backups can restore the customer & corporate data. Anything less is failure to practice Due Dilligence & leaving themselves wide open to having a lawsuit shoved up their arse.
Heads will roll & the lowest folks on the corporate totem pole will get thrown under the bus when this debaucle comes back to maul their arses.
"What person smart enough to make backups in the first place fails to check to make sure the backups are useable?"
A fine sentiment - but it's no use checking the first few backups - you have to check every backup to be certain that it's all working correctly. Of course nobody does this.
It make more sense to maintain multiple backups via unique services/methods in the hope that not all of them will fail at the same time.
"nobody does this"
We do. restore production backup, anonymise it appropriately and then use it for integration testing. Handy for performance tests and data dependent tests when you don't have sufficient unit test coverage. The testing of the backups is an added bonus
Does require sufficient hardware so is not free.
We also do proper bcp tests but those are months apart.
What person smart enough to make backups in the first place fails to check to make sure the backups are useable?
Earlier this year I setup some geo-redundant web/database servers for a somewhat large outfit to use for some fairly mission-critical stuff. I asked the guy I was subcontracting to about backups, he told me "that's Muppet's problem". "Muppet" is the name given by the person to whom I was subcontracting to person the whole thing was being handed off to.
Given my experiences with Muppet I would say that the chances of there being anything even resembling a functional backup at this time is around zero.
I'm doing a migration tomorrow. First step is always: backup production, restore to test, run the migration there, validate the results. You now know you can restore the production database and you've seen your upgrade or migration run with the current production data. Only then can you proceed...
OK maybe they didn't check backups regularly. But if they were actually going to go live with them, then FFS check them first, before you wipe the originals. (BTW maybe I'm missing something, but since when were backup files used to replace live ones except under emergency conditions anyway?)
If you willingly place "important" (by your measure of importance) data into the hands of a third party with no private copy of your own, you can scream and cry and gnash your teeth, but it's pretty much your own fault it is all goes boobies in the air. You can choose to make backups, or not. However when it's all out of your control, the integrity of your data depends upon the competence of other people, and you'll generally find the less you pay....
I just wonder how many people or moaning now, but will continue to use the app regardless. Until the next time? Or the time after that?
I work for a video production support company and having more than a single backup protocol is useful. I typically run a daily hardware duplicate of production drives (or media spaces) as an Online backup and then use drives or tape as appropriate for historical Nearline and Offiline backups (which include archives of changes daily). That way, if one doesn't produce the expected results, the secondary or tertiary backups will usually have the required data.
So far, I have been able to find data that has existed in our systems for at least 4 hours (backups are run frequently) in one of these backups: Online duplicates (most common and quickly restored) for files that were accidentally deleted , Nearline backups (with trailing archives of changes for the last 14 days) and Offline backups (typically completed projects). All of this is scripted of course, but must be monitored. Another bill to the client.
It is expensive, requires maintenance and vigilance, but we find that we only very rarely are at a loss to restore data from servers or workstations. This requires massive storage, both on and off site. Not for the light of wallet.
It is also a service and its cost is easily passed on to clients who may not have digital archives of the work we have produced. It is a moderate moneymaker in this respect as well.
If your data is worth something, back it up two or three different ways, so you always get a backup somewhere, even if one backup method fails. You will be thanked one day for this.
Anon cos wife works in EYFS.
I asked herself about this.
She, and as far as I'm aware, all local schools around where we are use a suite called tapestry.
It allows schools to download their data and burn it to CD - who knows when or if they do.
Though interestingly their website says nothing about data integrity.
I was asked to look at Tapestry a couple of years ago, after reading everything I could find about it, I recommended AGAINST using it, as there were big issues over how secure the data was.
Also, the data was going through/stored on a US server, so (once again) technically broke UK data protection act (this might have changed by now).
Same anon as "I told you so".
BTW, I set up an offline data back-up system for them that needed a single button push to launch, and a supply of SDXC cards to use; they hadnt run it once, the last time I checked, and since their "IT Manager" doesnt even know how to set a PlusNet router up, I doubt it is doing anything but gathering dust.
We are reliably informed that the IT-technical term for erasing a working system before the new system is up is "hubris".
Even at home, you replace a hard drive long before you expect it to fail, because new hard drives are quieter / more capacious / more reliable than the one you're using. Then you put the old drive on your secure shelf as a known good working backup. In addition to any other backup systems you have in place. It's "free", or nearly so.
"already going to have weeks/months of work to redo without not being able to do my current work too!!"
Even though they have been let down by them this person was saying they were annoyed as they are currently unable to use it for their current work!
That's like being thrown off your horse and in the process the horse injuring itself and then moaning about not being able to get back on the horse due to its injury.
or you outsource it...
It just takes the manager to take it seriously. Sadly like many other industries the education sector is run on next to no budget and people with the skills to make decent suggestions are often over-ruled.
I used to support a handful of schools, nurseries etc from a MSP in east london, we knew their budget was limited so offered a discount but treated them like any other corporate when it came to their data. They may have gotten a slower response time but they could live with that.
There is always a way to do IT properly - you just need to plan it properly, and if you don't understand it listen to someone who does.
At least the data is safe now, in the clouds of heaven.
Suggestion for press release:
I know how much your loss must grieve you,
but here's a thought I'd like to leave you:
Although you're sad, please enjoy the consolations;
that now god is reading your observations.
"the Cloud" is just a fancy name for somebody else's computer/network/SAN
The problem is, especially in IT, is that if you don't pay the peons their worth, they will start to skip on some responsibilities, especially should you start to assign more and more responsibilities to the same person without proper renumeration (salary increase, extra perks etc).
Or you don't want to pay $$$ for a proper test/lab network with which to test the backup/restore functionality of your data. Backups cost money, and having a test network is even more of an expense, but at the end of the day the expense of having a test/lab network will look like chump change should you experience total data loss and world+dog is taking you to court. Over a barrel.
I'm shocked I haven't come across this product before it's just the sort of crap I usually find out about a month after the contract has been signed then I'm expected to retrospectively approve the non-existent security and compliance on it.
Then I'm the bad guy not the fucker that didn't follow procurement and breached the DPA.
I wonder if anyone will sue the company or customers. Clearly the loss of data is a breach of Principle 7 and vulnerable individuals have potentially been placed at harm because this data is no longer available.
Thanks for the story, it'll go in my bag marked "Cloud isn't always shiny and cheap"
I run, for a much-reduced fee, websites for a few charities. I tell them I am covering 'the usual back-end admin' and don't ask them to make decisions on this. If I did, they'd try to cut it out of the budget, which I would not care about (I am already making a wee loss, but my work for them is my donation), but which they eventually would care about. So they get all the patches, updates, etc, and they also get three back-ups: local to a HDD on their premises, one to a server on my premises, and one to an external cloud server (based in EU). Luckily, disaster with data has never struck, but I don't ask them to be wiser than they are about security and we are all happy.
I remember a few times as a breakfix engineer going on site and whilst it wasn't part of the job, I would always ask what backup they are running and just have a peek. Number of times it had failed was quite surprising. Usually due to capacity issues or changed paths / shares etc, but a classic was a FAT formatted USB HDD. All was fine until the backup hit 4Gb as a filesize - pretty much a week after being set up.
Unfortunately there is a mentality that "the cloud" is the answer to everything and magic just happens to keep your data safe. The people the do not understand the difference between availability and backup is staggering. Couple that with the Apple connection (yes I know the only made it available) but it on the App Store so it must be good and "just work".
The company behind this that were stupid enough to wipe without testing deserve to go out of business. I would actually go as far as preventing them from putting anything back into the App Store (or Google Play).
Biting the hand that feeds IT © 1998–2020