I always smile when the statement generally goes like this
Affected a small number of users..
Neatgear has cocked up its cloud management service, losing data stored locally on ReadyNAS devices' shared folders worldwide – and customers have complained to The Register about only being informed four weeks later. This week, the San Jose-based networking business sent an email to customers, seen by The Register, confirming …
I was affected, but didn't report it because:
A) I initially thought it was something I'd done.
B) Only when I got an email today did I realise Netgear had screwed up.
C) I had backups - so no data was lost. I've been burned before by a ReadyNAS failure.
They can't claim "only a handful of people were affected" before they've even told them.
Netgear routers and switches are mostly excellent but their NAS division is a mess!
They even all of a sudden dropped support for many NAS models to release new ones trying to force customers to trash the working ones for new products just to get updated software and features.
The best NAS come from Synology with top-notch software support that lasts many years.
The best NAS depends upon features you require.
I think Synology are top when it comes to software, however you can get an x86 equipped Qnap which enables faster transcoding for things like Plex and it doesn't cost to much more than a Synology NAS. Both have regular software updates. I have a Drobo at home though, the hot cache won't take up a HD slot and has an inbuilt battery. Software is forced through a windows client which makes it clunky and I admit its priced to expensive (I would prefer Drobo brought out a 2 bay and a 4 bay network unit instead of sticking with 5).
*Edited to correct Engrish
I've had many issues with a number of Synology devices and versions. The underlying disk software is unstable. It can easily freeze all disk operations, especially when dealing with iscsi/luns. I never do any volume or disk changes on our current device due to the risk of the lun being dropped and disk operations freezing.
Combine this with their HA operation which will switch over on this instance to a backup unit and corrupt real-time data in the process. You now have to decide whether to kill the old active server completely and hope it successfully re-synchs. The data is corrupted anyway and any large dbs are now lost the fun world of transactional recovery.
Support is (was?) only available as a free option - great, apart from the response time to a significant issue is now days, with a few minutes access every now and then over the next couple of weeks before declaring that they can't help.
The device are a good price, they should provide a good option for a reasonable priced (fairly) HA solution but they really struggled with the basic disk operations and data corruption, and no option for premium support makes them unusable for anything other than a file store.
> The best NAS come from Synology with top-notch software support that lasts many years.
To be blunt, both Netgear and Synology have similar (not too bad) reputations for home / prosumer NAS gear. This event isn't going to be helping Netgear though.
My preference is FreeNAS as they're based on tech I like, I'm comfortable choosing my own hardware + building my own systems. And they accept Pull Requests on GitHub. eg to support more hardware, add more features, etc.
But, horses for courses, etc. :)
I bought one of their Gbit switches because it looked good value and it's fast and reliable.
Relying on Cloud can increase chances of failure, data loss and security failure, and Netgear really F'd up even thinking of coding a local delete after loss of service, given even a local issue could cause loss of cloud access!
FreeNAS here too. I looked at many off-the-shelf NAS earlier and the cost per capacity was ridiculous for RAID1 and RAID5 on pathetically underpowered 100Mbit Ethernet hardware, let-alone for Gbit Ethernet! FreeNAS with commodity parity RAM server boards is much more robust, flexible and fast, and can be much better value for several disks; FreeZFS makes all Windows and Linux filesystems look very dated and fragile!
> Can I ask what case you used (Assuming hot swap trays and small form factor)?
Personally, I didn't go for either as I have several unused full size tower cases already (big Coolermaster things from years ago) so just reused some of those. And my use case doesn't need hot-swap drives, so didn't bother. ;)
Other people definitely go for the small form factor thing though, so there's plenty of discussion around that kind of thing on the FreeNAS forums.
If you haven't seen it already, the "Hardware recommendations guide" is useful:
That gets updated fairly often - every few weeks it seems - unlike what the main text on that page says, so it's fairly up-to-date. :)
Slowly but surely, people will start to get the message : cloud is NOT backup, despite all the assurances to the contrary.
The ONLY backup you can be sure of is the one you can hold in your hand and have tested to be sure.
If you're not storing your backups offline yourself, you're just the next victim waiting to learn how it happened.
It sounds like they had a brilliant design where the local ReadyNAS you can hold in your hand would wipe itself if the ReadyCLOUD account was marked as closed. So if the cloud server makes a mistake and decides the account is closed and wipes the data and then the local NAS gets told that the account is closed, then the data is wiped everywhere. If it really does work that way, then it is a mindbogglingly stupid design. Essentially you are not in control of your local NAS at that point and it can only be considered a backup device, not primary storage, which it seems a lot of people thought it was with cloud backup.
Slowly but surely, people will start to get the message : cloud is NOT backup, despite all the assurances to the contrary.
Actually it is. I could put an axe through my photo drive right now, and I know that every byte of that data is available on a cloud server.
What's more, I can go to the place where the server is and copy the data at local network speeds to a new drive OR I could go a little further and actually plug my new drive into the server and copy the data over at USB speeds (USB2 since the machine is an older machine).
And the beauty of being cloud-based means that when I add pics here, or other data to the non-pic folders, it's backed up to an offsite location (so if I set fire to the computer and lose the house as well, then my data is still safe).
But as said, this is my hardware and my configuration. Cheap and easy to build (if you can afford a Pie and a large enough USB drive to plug into it, you have the hardware you need), and as reliable as I am. Also running some calendaring stuff on it as well, which updates through the somewhat crappy Lightning on Thunderbird (keeps pushing notifications of past events), the calendar tools on Evolution and also whatever is on my mate's Android devices.
Cloud can be good, but keep your local copies backed up where ReadNAS can't delete it for your if it can't talk to their server. Or someone malicious at your cloud provider, internal or external hack.
"We have already identified the root cause in our server software and applied a patch immediately after the incident occurred".
I'm not sure if, reading that, I'd be reassured that there'd be no more problems or hardly any more problems or if I'd wonder why a problem that could be fixed "immediately after the incident occurred" could have been anything but obvious on cursory examination. I might wonder how many more such "immediately" fixable faults were overlooked by similar lack of cursory examination and remain unfixed.
Why would anyone trust important data (with no local backup) to this Wild West world of the cloud? Surely we must be years away from reliability. If the IT industry's track record on quality is a guide, it's more probably decades.
"Surely we must be years away from reliability. If the IT industry's track record on quality is a guide, it's more probably decades."
Years away? Looking to the future, or to the past (or both)?
Maybe you would like to amend that to read "If the IT industry's *recent* track record on quality " (recent = a decade or two)?
IT existed before Windows and much of it was trustworthy (some of it still is).
Trustworthy IT will also likely exist after the decline of Windows (though Android isn't exactly setting much of a trustworthiness example so far, and nor is the InterwebOfTat, even without MS. Same can be said about "professional" services outfits like Crapita too.).
The IT industry and its victims currently seem to be in a Dark Age where the industry (OK, the Windows ecosystem) has managed to set the expectation that trustworhiness is neither possible nor relevant. Consequently customers and users (not always the same people) are suffering as a result, and not realising that there was and is another way
The following words from Doctor Syntax three days ago highlights how low modern expectations have sunk - people are no longer able to take for granted the ability to reliably get the right stuff on "disk" at the right time (or to reliably know that it has failed, so recovery measures can be taken).
Thumbs up for pointing out the elephant in the server room (whether it's an on-premises server room or some cloud providers/outsourcers server room is irrelevant).
"[...]If you're putting five nines before everything else you're worshipping at the wrong altar. Consider the following:
Maintaining integrity of the data you've got.
Being sure that new data gets added properly.
Being available to add new data.
Availability is a poor third there. Of course five nines availability is something manglement is able to understand and get fixated on. But if you have a big data loss you'll probably lose your five nines whilst you recover it and if you don't recover it all your five nines during the time you were acquiring it turn out to have been a bit pointless. [...]"
Why would anyone trust important data (with no local backup) to this Wild West world of the cloud?
I thought NAS was a local copy? So that would mean that Netgear is being kinda nasty, delete the account from their remote services and they delete your data.
That said, they do say it's an enterprise account, and with that in mind I can see the "value" in making sure data is wiped, but still. They should leave data removal on local devices to local admins, and only delete what's on their own servers. Or do they have a MS-like clause ("We own everything, you only rent it for a time")
Exactly. If what actual "cloud" and "enterprise grade", then the data would not only be properly backed up, but a single server outage should not affect the operation at all. AT worst, users may notice a slight increase in latency as everything switches seamlessly over to a different, unborked bit barn somewhere else in the world.
I don't care if it's free, cheap or expensive, if you're going to call it "cloud" and "enterprise grade" then it fecking well should be that.
"if you're going to call it "cloud" and "enterprise grade" then it fecking well should be that."
That's a very quaint (20th century, even) viewpoint.
Don't you people realise that bullshit is what counts these days, be it in IT, politics, or a Billion other places where money changes hands in large quantities?
> I don't care if it's free, cheap or expensive, if you're going to call it "cloud" and "enterprise grade" then it fecking well should be that.
Note they didn't say *which* enterprise.
If they'd been clearer, saying it's of "dodgy enterprise" grade (such as IBM or Capita) maybe their customers would have known to avoid it?
"Compounding the issue, as part of a clean-up process, Netgear decided that when a ReadyCloud account is marked as closed, the NAS holding that account's home folder should be deleted along with all of the data it was holding."
I can't understand why a local hardware device holding your data gets deleted just because you no longer have a cloud account, why would this ever be necessary?
"Should note that ReadyCLOUD is an enterprise VPN grade remote access solution and at no point has it ever been compromised."
Yes. Because no one ever compromises a system and wipes out data. Your product may not have been broken into, but your product was broken and that means your users have lost data.
Normally I'd tell people to make sure anything that they have stored "in the cloud" they have a local copy of as well, but this has taken out the cloud and local copies. And worse seems to have done so in a way that's not easily recoverable. I advocate use of some "cloud" stuff for backups (as I've said here before, owncloud to keep some of my stuff synced between home and elsewhere - my hardware etc though) but this is a risk factor that I never really considered (or actually knew about, since with other cloud providers (Mega, Owncloud, Box and Dropbox that I've used - oh and the old copy.com as well (Barracuda IIRC) when you delete the account they delete the data from their servers, but the data on your disk remains untouched until you decide to delete it.
Who messed this up so much that a "server outage" could wipe user's local copies? Perhaps someone needs to be having a deep and meaningful conversation with the dole queue?
"We are currently working with each impacted user to recover as much of their data as possible using custom data recovery tools"
Why not common recovery tools, long tried, tested, and proven? (Honest question, maybe they've done something weird with the way they store data, in the name of "data security" (see above about compromise and wiping out data!)
No icon does this justice. Even as a non-user of their products but as someone who has had to tell people that their pics are gone forever (sometimes involving a loved one who's died), this is quite an annoying situation.
But kudos for working to recover data rather than "Our T&C says we're not responsible, so bye bye" like so many others do.
Biting the hand that feeds IT © 1998–2019