DigitalOcean drowned my startup! 'We lost everything, our servers, and one year of database backups' says biz boss Nicolas Beauvais, the CTO of a two-person AI startup called Raisup, raised a ruckus on Twitter on Friday to revive his company and it worked. Cloud hosting biz DigitalOcean, he declared, "just killed our company." But with the help of the Squeaky Wheel support tier – by which we mean making a lot of noise on social media – it' …
For non-businesses a simple monthly coping the data over to a CD will work.
What's a "CD"?
Oh, you mean those funny little disks that could be written and re-rewritten, but the likelyhood of reading the data was perversely inversely proportional to your need?
Doesn't "CD" stand for "Corrupted Data"?
I'm similarly unimpressed with their competence, and I have little sympathy, but saying things like "they deserve to fail" is victim blaming. They don't deserve to fail because they did something wrong. They deserve to have their problems pointed out to them rather brusquely and to lose business if they don't fix them immediately.
"It's only amongst university students and Liverpudlians that victim status automatically absolves all sins."
When is it a sin to attend a football match and *expect* that you would be able to return home at the end !!!
Yet another blind bit of tribalism that of course excuses *any* insult .... even to the dead !!!
You probably couldn't have illustrated my point better, you know. Only a Scouser could assume that the dreadful events at Hillsborough exempted the entire city from all criticism for all time.
By the way, were Juventus fans allowed to expect to survive their trip to Heysel Stadium?
"You probably couldn't have illustrated my point better, you know. Only a Scouser could assume that the dreadful events at Hillsborough exempted the entire city from all criticism for all time."
Nice assumption ..... but wrong !!!
Also your criticism *includes* the dead which is my point *not* excusing anyone .... although I would like to know what these sins are that excuse attacking the dead.
I also don't understand the relationship between Heysel & Hillsborough ..... both were bad and one does not excuse the other or even 'cancel out' the other which appears to be your argument.
The main sub-text is that you don't like 'Scousers' and object to any sympathy someone may give the impacted families. As I said 'Blind Tribal' !!!
There is a point where Tribal football fanaticism ends and human decency starts.
Some (most? all?) cloud providers charge for data egress. If your static data (that you need to backup) is large enough, that can be a significant (crippling) cost, particularly if the backup cadence requirement is regular.
So most cloud provision strategies work off multiple regions of the same cloud provider and the bean counters have calculated that the risk of fail of multiple regions is low. Bean counters, though, don't have a function for "account locked", which is a non-trivial risk given stories like this one.
Data egress is a cost of doing business (safely). If your team are aware of the consequences of provider fail (even aws and google have cross-region outages that last several hours, and some businesses that hour may actually be when they make the month's revenue) then they may be able to design some parts so the data is mirrored directly to the DR backups, rather than into one cloud then out to another, or efficient deltas transmitted between sites rather that who files/databases/object/etc every time. If none of those strategies apply, you should probably make your customers, investors and BC insurers are aware.
Data egress is a cost of doing business (safely).
Lots of things are 'costs of doing business'. Most businesses don't get all this stuff right, legally or morally (some will do one much better than the other of course).
For a smaller business every penny can count, and sometimes things have to be cut. Hell, just look at the efforts Google, Apple etc take to avoid paying their fair share of taxes, or to provide decent working conditions for their staff. Look at all the "intern" crap that goes on in the "land of the free" where they've made an artform of not paying people a decent wage (though the rest of us can hardly argue!)
Big corporations with large highly-experienced legal and technical teams don't get it right so why should the rest of us be held to a higher standard? Other than for those of us who value integrity and customer service of course.
Does your own business get everything perfectly right?
These guys have done well, even if they had trust/performance issues with their provider.
Sure, sometimes every penny counts and sometimes things have to be cut. And sometimes DigitalOcean terminates your account and your Fortune 500 clients get angry at you.
There is only one standard that matters, and it's business success¹. Generally when you lose all your customer's data, this tends to end any hope of business success rather terminally.
¹And the law, I suppose.
Sure, sometimes every penny counts and sometimes things have to be cut. And sometimes DigitalOcean terminates your account and your Fortune 500 clients get angry at you.
Yup. Sometimes the building burns down. Sometimes the neighbours building burns down, and you're not allowed in to work due to #hazard. Sometimes someone drives through a power pole. Sometimes the local substation suffers 'thermal runaway'. Sometimes natural disasters occur. Sometimes the PFY or the PHB types something they shouldn't and hoses the database. Sometimes someone lets a nasty worm into your system. Sometimes Win10 decides to install itself and makes your machines unusable.
There is only one standard that matters, and it's business success¹. Generally when you lose all your customer's data, this tends to end any hope of business success rather terminally.¹And the law, I suppose.
Ahem... MS. FB. Google. Yahoo. Most large scale email providers. Western Digital. Intel. Most ISPs. Many of them are still doing nasty stuff to customer data and still getting away with it. Can you name any large company that handles a lot of customer data that hasn't been responsible for massive losses to others? I only named a tiny portion here, but I honestly cannot think of any firms that haven't had bad data losses or caused significant productivity losses for numerous clients.
They're still thriving. I also know of several small firms who did OK despite big screw ups - sometimes that's the only way to teach management the importance of regular backups, when even losing a day's invoicing data is a big problem.
I see DigitalOcean's IP range regularly show up in my 404 logs, so I'm intrigued by their aggressiveness.
From where I sit, they don't seem to do much about people scanning the Net. That said, it's nowhere near the scanning pollution that OVH represents, but stil, they're on my radar.
"From where I sit, they don't seem to do much about people scanning the Net. "
Indeed. The people doing the scanning are bad enough, but the fact their hosting providers can't be arsed to do anything about abuse reports is unacceptable.
If IPv6 ever happens, at least that should bring an end to all this scanning nonsense. The address space is just far too vast.
@cbars
There are 2^32 ( 4294967296 ) IPv4 Addresses.
The "real" number of IPv4 addresses on the internet is actually lower than that due to unroutable ranges (RFC1918) and other carve outs.
Meanwhile, in the land of IPv6, just one little /64 subnet contains 18446744073709551616 possible addresses, which is already significantly more than the entirety of IPv4 ! Looking beyond /64, you've got the typical /48 "site assignment" (e.g. assignment from ISPs to customer sites) which has 1208925819614629174706176 possible addresses. And finally, ISPs start off with a /32 allocation each which is 79228162514264337593543950336 addresses per ISP.
"It happened, and is hasn't."
Hey Smart Alec (a.k.a. the hatter)
I think the point being made was perfectly clear.
"when IPv6 happens" was clearly meant to imply "when 100% deployment happens, not just a small number of obscure ISPs and an even smaller number of websites that can be bothered".
not just a small number of obscure ISPs and an even smaller number of websites that can be bothered
The 90s called and want their meme back !
In the UK, most (all ?) of the largest ISPs do IP6 by default - BT Internet, Sky, and IRC TalkTalk who between them have the bulk of users. So you can untick "obscure ISPs".
And I think you'll find that Gobble and FaecesBorg aren't exactly obscure websites either.
Google reports https://www.google.com/intl/en/ipv6/statistics.html that currently about 1/4 of all it's traffic is native IPv6 - so clearly more than a handful of obscure ISPs with few users. If you look at their per-country stats, the UK is at about that level (24%), while Germany hits 44% and Belgium hits 54%.
Facebook reports https://www.facebook.com/ipv6/?tab=ipv6_country higher figures, and interestingly a different pattern - and the USA showing 58% IPv6 adoption.
I'm curious about the supposed roughly 1/4 of US-to-Google traffic being IPv6 - is this pretty much just mobile users? Is there only one AT&T Mobile-to-Google link, and it happens to be IPv6? Because short of renting an instance on AWS to play with, actual IPv6 is pretty darn thin on the ground over here (i.e., I've yet to see any in the wild, not on a private LAN in my house or my AWS instance).
I'm sure the stuff exists. Hurricane Electric was advertising it, what, a decade ago? I just think it's funny that I've seen more Space Shuttles than I have IPv6 requests that weren't just me talking to myself.
(Icon because I'm basically clueless)
In the end it doesn't matter. If people seeking to find devices to abuse have to wade through a vast ocean of addresses, they just make a bigger botnet.
The sheer size of the address space may make it harder to find a specific device on an address, but it won't stop them trying.
After all, such people are not using their own resources to do the searching.
They don't have complete visibility into what users are doing, as users have private server space. A script can identify things like serially accessing IP addresses, but it would be harder to detect other types of automatic scanning. For example, I run an endpoint for a VPN on a DO-provided server, and it also makes lots of HTTPS connections to sites because it handles my traffic and those of a few friends. The only way to tell this apart from a scanner that is a bit random with the servers it attempts to access is to log my sites and build up patterns that would show that I'm just browsing rather than scanning. If they did that, I'd be a bit unhappy about the privacy implications and I'd probably move my endpoint.
However, they should really check some other types of scanning, like attempts at SSH logins. I see these from lots of places, including Digital Ocean addresses, and those should be very obvious from the traffic. I simply set a rather vigorous fail2ban config, where you get quite a few attempts at first but get locked out for a rather long time (currently 48 hours) when you exceed them. Previously, I saw bots start trying again after a delay, but they usually give up if their retry attempt is likewise rejected.
"I see DigitalOcean's IP range regularly show up in my 404 logs, so I'm intrigued by their aggressiveness."
I suspect the DigitalOcean IP's scanning your address ranges may no longer be fully in control of their contracted owners. Cheap hosting doesn't necessarily correlate with well managed servers and many sins are committed by well meaning web developers.
In terms of the why DigitalOcean detected this, I suspect it was from high server utilization (i.e. caused by the python script) rather than traffic analysis which is where your observations differ.
None needed ahole ; GO look at your security logs and get back to me
Done. All 3 of them. No problems. Same for the vpn, email and https systems.
One box did show an odd level of traffic, but that was due to a patch cable in need of reseating - traffic now back up to it's normal levels. One site did get an outage sat night/sun morning but I suspect that was upstream. Will check with the isp later this morning. And will noan as they had no manned call centre yesterday. Faults do happen on Sunday!
Regular stuff in f2b logs, but that tailed way back when I moved away from the standard ssh port. Now instead of several hundred strikes per hour I'm lucky to see 2 in a week!
Oh yeah, tell me about those imaginery scammers again? What was it you were insisting is there? Or were you just blowing smoke out your arse coz someone called you on your bs claim? (I'd've been happy with "most" instead of "all" - expect "most" have some level of badness)
I'm afraid I have to agree with him - most don't even bother to reply so I tend to add a delivery tracker to the email.
Where I did get a reaction was when I CC-ed local law enforcement. Either that worried the ISP or the cops were bored, but I had some 100+ entries in the 404 logs with a whole lot of access attempts so I decided to throw a brick in that general direction. It was interesting, but I'm not sure anything ever came of it.
I'm afraid I have to agree with him - most don't even bother to reply so I tend to add a delivery tracker to the email.
You've never emailed me (nor has anyone else) so perhaps 'every cloud server' doesn't host scammers etc. Some, maybe most - sure. But every one? Are you sure? What's your test method - there are literally millions of servers out there!
No, what you have to explain is how is it that you have Fortune 500 clients and yet you have no copy of your data in another place than your hosting service.
I would hope that this will be a lesson to not put all your eggs in the same basket, always have a backup plan, etc, but I'm guessing that you'll stay just as stupid and leave your entire company in the hands of people you don't know until everything crashes on you a second time.
Then you'll be whinging that life is unfair. It will still be, ultimately, your fault.
well, at least it appears that it STILL HELPS to GO PUBLIC when normal channels fail to get the needed attention...
and that the ultimate problem is bureaucracy and bureaucratic attitudes.
but yeah, having a local backup can be your ONLY salvation. a little over a week ago, a different contractor was supposed to commit his changes to github so that I could integrate it. If it wasn't bad enough that this guy took 2 weeks to do 2-3 days' worth of stuff [my opinion, or how long would it take ME to have done it], and we'd been after him the entire time to at least commit SOMETHING to the repo, he managed [with some GUI tool on a Mac] to WIPE OUT all of my significant changes and additions over the last 2 days when he uploaded his stuff to github (in effect, syncing up his private repo with the one on github, literally). ONLY because I'd had the foresight to have done a 'git pull' on my home machine before leaving, and having the ability to ssh in and grab a tarball of what the repo was 2 hours earlier, was I able to quickly fix this. a simple 'revert' probably would've left things in a worse state than a) tarball a snapshot, b) re-do the 'git clone' c) untar everything I did, d) git commit+push, and THEN have the other guy (one at a time) check in the files he worked on. I think that my method was faster, at any rate.
So yeah it helps to have some LOCAL backups, on multiple machines, just for things *like* this.
Oh yeah that particular contractor was given the "thank you very much for your hard work and effort" letter. I won't go into the other drama that preceded it, involving slack and a literal 'commit war' over a 3 day weekend when way too many things needed to be ready on Tuesday AM for review, 99% of it my effort for "last minute, because, THAT guy" integration purposes.
(ok vent complete)
Sometimes. SOMETIMES. You get lucky, and that destructive sync still leaves enough info in the reflog to roll back the head to the right location.
God knows I had to deal with self inflicted versions of such a decade back...
Now I (sadly) have no version control whatsoever, except daily backups and "I should save a copy of this and bump my minor rev number..." to keep me sane. Hah.
Gotta love proprietary binary save formats, compressed single file XML bundles, and fishing the records room for the ONE licensed copy of xyz.abc.soft from 1999 that will read that format... Then finding it needs NT4...
Then finding it needs NT4...
You're in luck.
I was helping a mate clean out his garage on the weekend and found a virgin NT4 install CD. And some old boxen it'd probably run on.
I know exactly where it is.
Well, that is if the recyclers haven't yet processed it and shipped it all off to wherever they send it to. Better be in quick if you want it so first thing in the morning I can try to stop them :)
I was lucky, it whined, moaned, and May'd about not having NT4, but ran in a VM of Win2KPro-SP4.
And to my horrors, I got yet more frustration, because the project file seems to be intact, but it's all setup for a different PLC than ieverything was originally built with.
Should be Siemens over Modbus, looks to be AB over DF-1.
I smell some JP-8 (OK, Jet A-1...) in the future to sort this mess out.
Icon because I couldn't find an archeologist.
That begs the question of why such a small startup has such big customers so early on, but that's not really relevant
I can answer that...
We (my last company) had the right tools at the right time at the right price, and were able to convince the customers that we were able to do the job. It did help to have an historic tie-in with a long-standing and rather large manufacturing firm, but that only got us to the meeting. It was what I could show them we could do that got us the gig.
"To be fair, it *is* a two person company. They're probably working out of shoebox with no proper AC/power for a server"
They're a two person company relying on a third party to provide their infrastructure because they can't do it more cost-effectively themselves. They may have had a tested, sensible DR plan and appropriate site redundancy with their hosting plan, but weren't prepared for an "account suspended" incident in much the same way as a landlord locking out a business tenant and shutting off the power would affect a similar physical server installation
In all likelihood, being in their first year, they likely choose Digital Ocean based on their previous experience and moving to a more appropriate provider maybe on their to do list as their sales increased.
"the potential brand damage to his cloud biz"
How about potential brand damage to cloud biz in general? It's another illustration of the dangers of relying on other people's computers. Irrespective of whether or not Raisup had their own backups they're still going to be off-line for some time until they're restored and tested elsewhere and their customers pointed to the new location. And all because of of an script run by a hosting co. The poor response might have been specific to DigitalOcean but the risk of a hoster's automated whim must surely be generic.
A start up two man band with Fortune 500 clients? I guess that explains why the backups resided in the same datacenter as the production systems. They were busy grubbing the money, instead of making sure they had the basics covered. I hope this episode gives them a clue as to the "1" in 3 2 1.
I do wonder what backups can do for AI, how would that process even work?
All providers keep an automated eye on resource usage for any shared system, goes with the territory and is generally a good thing. Further down the food chain however especially with budget & free web hosting providers yes some do go nosing around your file system looking at code and content.
If you read the thread, it wasn't the python script per-se that DO decided was malicious, but their workflow itself.
To run that python script they spin up ~10 new instances to run 10 copies of the script in parallel, and then once it's done the instances are killed.
DO decided that was unusual and locked out the account, asking for more information on their workflow to re-enable: Screenshot.
Which, in some ways, is actually worse than "we thought your python script looked dodgy".
One of the commonly touted "benefits" of cloud is that you can spin stuff up when you need it, and then kill it once the task is done. Which is exactly what they seem to have tried to do here, except that DO decided that was suspicious...
What I find worse, though, is DO's approach to it. I can just about buy that it's a change in behaviour, and might be suspicious. But, why in the name of all that's holy, would you lock a customer's entire account out because you don't recognise the use-case? Maybe at a stretch, kill the instances that are acting suspiciously, perhaps prevent creation of new instances, but don't fuck with existing instances that aren't exhibiting the behaviour you're concerned about.
That's not a customer service fail, that's an Ops fail. Do not fuck with production, especially other people's production.
I run some stuff on DO, but I do also have provider redundancy in place with automatic failover. I think I'm going to need to double-check that that's working as intended, and maybe move away from DO completely depending on what their post-mortem says once it's posted
"That's not a customer service fail, that's an Ops fail. Do not fuck with production, especially other people's production."
That's probably not how the thinking works. Your production - that's what pays the wages. Other people's production - their problem.
The post-mortem should be interesting. If it's just "Our checks had a bug that did such and such instead of that and that. We've fixed it so it won't happen again." that's not very good. The real question is why it was even possible to implement something that would cut off a paying customer without intervention.
It goes back to the comment I made on the Martin Fowler article. It's all very well being a fast mover and fixing your errors so your quality improves quickly; when there's an error it can do substantial damage in the course of discovery.
Hmmm. I wonder if DO might not mistakenly have stepped in from the opposite side of the mistake, trying to spare the customer grief.
We hear many cases of cloud auth compromises where the hapless customer is left with $$$$$ bills of cryptomining or the like.
I have a very low $$$ threshold limit because my VMs just sit there at a predictable monthly rate. Past that I get an email.
But if DO had hijack safeguards, perhaps incorrectly defined/parametrized, they might have been thinking this massive spin up was a sign of credentials compromise.
Even freezing backups can help there. Remember the 2-3 stories so far of cloud hosted companies whose backups were terminally erased after key losses? One more reason to back up elsewhere.
Yes, interested in what debriefing will have to say. Especially the “what procedures does DO have to put someone on the phone that can fix things (but not be social-engineered)”.
nice clarification - my skim-reading of the article missed that aspect [i.e. the 10 parallel jobs] and since THAT was all it was, D.O. should've NOT shot themselves in the foot like that. If they don't want people spawning parallel jobs, they should set some kind of limit on that kind of thing, total CPU percentage, total number of jobs/threads, yotta yotta.
(effecitvely) banning them entirely was a HUGE mistake.
ideally these cloud providers might be creating the "lack of backup" problem by CLAIMING that you have 100% reliability "in the cloud" with backups and reliability and/or some other such thing... until their bureaucracy SCREWS YOU.
but yeah us old schoolers with 20:20 hindsight can easily say "you should have had a backup"
Yet another example of why cloud computing should not normally be used for business critical applications.
The good use cases for the cloud
1) Short term processing peak
2) Web server
3) Running test (and/or development) servers well away from production systems
4) Running production while waiting for new servers to be installed
Running routine production long term in the cloud is NOT a good idea. Given the cost of cloud computing any sustained load will normally be far cheaper to run on own equipment over a 3 year or longer period.
The effects of GDPR and the US Cloud Act should also be considered for any company in the EU considering having any personal data in the cloud.
User pays an initial fee (~$10) by credit card to gain the attention of human support staff, the level above the off-shore script followers.
Company refunds the fee once they've hoisted aboard that there's a real issue that requires their expert touch to be solved.
Many would willingly put down thousands of dollars just to get past the useless phone menus, on the assumption that they'd get their fee back later once they'd proved the issue.
considering that I'm willing to pay an extra 10 cents per gallon (sometimes, at discount stations) to use a credit card in a gasoline pump, just so I won't have to go stand in line TWICE to get gasoline AND fill up the tank AND get change back, assuming I handed the guy enough $ before I started, forking over a refundable $10 to get some human to fix your problems right away would be a GOOD thing...
it's like the phone company fixing my wires - which they've had to do several times - if the problem is inside the house, they charge you $75 for the service call. Thing is, every time it's been on their end. SO I'm willing to risk the $75 just to get the tech out to check the wires, which he generally finds are broken some place in the 10+ wire miles between me and the home office. My connection is LITERALLY at the "end of the wire". The neighbor's house has the wire literally heading the other direction.
So yeah. Good idea. If I ever get involved in tech support, I'll consider that option. Refundable $10 fee for accelerating the problem to a human's investigation.
"considering that I'm willing to pay an extra 10 cents per gallon (sometimes, at discount stations) to use a credit card in a gasoline pump, just so I won't have to go stand in line TWICE to get gasoline AND fill up the tank AND get change back, assuming I handed the guy enough $ before I started, forking over a refundable $10 to get some human to fix your problems right away would be a GOOD thing..."
Really? Pay cash in advance of filling up or pay EXTRA to use a card? Wow! Is the guy you pay hiding behind bullet proof glass too?
"Really? Pay cash in advance of filling up or pay EXTRA to use a card?"
Unfortunatley that's the way it works in the US. Cash in advance or card in the pump.
Takes Europeans by surprise the first time you rent a car in the US and go to fill it up. ;-)
Doesn't bother me these days though, since I generally refuse to go anywhere near the US these days.
When I used to go, I would still go pay up, since the card readers at the pumps were typically skimmer heaven.
What takes all foreigners by surprise is that you need swipe your card and enter 5 digit postcode instead of PIN (in fact most credit cards issued in the land of free do not have PIN code assigned by default and require one only for cash advance in ATMs). Mine has 4. I've heard about some hacks like enter 00000 but that never worked. But I never had any issues prepaying with card on the counter and getting difference refunded back to the same card, but yes, that requires two trips.
Well I am told you can enter the numbers of your postcode with trailing 0's to American fuel pumps
so when getting petrol at the pump in the USA with a British credit card, when asked for the zip code enter the digits from your postcode, ignoring the letters, and then enter zeros (0) to fill the remaining digits until you have 5 digits in total. For example, if your postcode in the UK is aa29 1aa enter 29100; if your postcode in the UK is a1 1aa enter 11000.
Not tried it myself yet . As only just learned of it …
as we normally play the umm try $20 and get surprised when it buys a tankful on a huge car when at home it would have tickled the fuel gage on our car that would fit into the boot of the average USA gas guzzler
"Doesn't bother me these days though, since I generally refuse to go anywhere near the US these days." - I went to the US twice, and that was enough thanks. In many respects it is just another shithole country, with badly maintained cities, junkies and homeless people everywhere.
'Backups' on the same cloud provider are not backups but just another copy of your data and they should have had backups stored elsewhere. I know this the hard way after I had 2 VPS with the same budget provider but in different datacentres both with the same data and when they went tits up I lost access to both instances. But luckily I had some backups downloaded locally but still lost some some data.
It did seem like a dick move from Digitalocean to not allow their client back into their account to at least retrieve their data though
No backup plan, no DR, no control of the servers that you need to run your business, no copy of your software you can quickly deploy somewhere else, no transnational persistence, and no easy way of explaining to your customers what an omnishambles this all is. Astonishing.
You will go bust very soon and probably blame Trump or Brexit.
No backup plan, no DR, no control of the servers that you need to run your business, no copy of your software you can quickly deploy somewhere else, no transnational persistence, and no easy way of explaining to your customers what an omnishambles this all is
Pretty normal business practice. How many companies own the building they're in? How many own spare buildings? What, no backup?
How many lease the servers that are in said buildings, rather than actually owning them? Leasing gives better tax breaks than owning after all.
Same for vehicle fleets - how many large transport companies don't own a single vehicle?
Besides, for a long time there have been businesses that offer 'total data solutions' from hosting to databases to backups. For the vast majority of firms this works well, potentially saving the company considerable amounts of money, time and effort. It's relatively rare that such things fail especially in such a spectacular manner - otherwise it wouldn't be newsworthy.
But yes, have your own backups (though ex-filtrating the DB data is another cost (either $ or data transfer allowance) - best find a way to update only changed records) and have a plan should things fail.
There are benefits to using large-scale providers. I used to do my own hosting. Keeping the website and emails servers synced wasn't too bad, mostly perfectly automatic but could take a bit of time if something broke. Failover wasn't automatic but only involved a simple tweak of DNS entries. But decent levels of backups was a different matter - storage is expensive for a small business especially with the levels of data we handled (full backups of customers drives, held for a month from the date the job was completed). At least cloud providers can get disks for probably less than 1/10th of what we paid. Hell, they could probably get a 32TB disk for the price we got a couple of 2nd hand 250G drives! And they (hopefully) have decent proven ways to manage your data and variations thereof, much easier and more reliably than you do.
How many firms could survive a fire in their basement rendering the building unusable for a number of weeks? At least if all your data is cloudy, you're safe from that!
In my opinion these are the sorts of incidents which should give startups a wakeup and reality check, not a podium to preach from. You get what you pay for.
Yeah I know. How DARE they not use unlimited funds when starting out. How DARE they take a provider's word for the promised level of service. How DARE they not re-invent the wheel and build everything else from scratch!
</sarc>
If you're experienced in starting a business from scratch, especially if you started it from a shoestring (I have no idea what their budget was), you'll know how easy it is NOT to get everything right - even things you know you should do often get deferred until you have the readies to deal with them.
data should be migrated to offsite backups. I use google APIs to backup to Google drive automagically.
system builds should be scripted and repeatable.
I'm a small businessman but when a web server got corrupted a month ago, I was able to rebuild and restore from scripts and offsite storage in under three hours.
data should be migrated to offsite backups. I use google APIs to backup to Google drive automagically
Be wary of any privacy laws you are under if you handle customer data on those backups :)
I'm a small businessman but when a web server got corrupted a month ago, I was able to rebuild and restore from scripts and offsite storage in under three hours.
I originally used "copy.com" to keep a couple of machines synced, but later went to my own Owncloud instance. I kept 2 machines running, a server in the office as well as a machine tucked away in a closet in a private home (with an ISP that quite happily gets out of the way of their customers doings). When the shop's phone lines went down during a flood, it was a matter of tweaking the DNS records to change the server. Same when the proper server's HDD suffered a sudden "grave spike in activity" as in some twat who was playing with cabling slipped and yanked the server cable, pulling it off its shelf onto the floor (the HDD didn't appreciate the sudden relocation).
I am quite happy for people to be using cloud stuff, but I do recommend you look to use your own first :) Given the number of services Google have suddenly killed of late, and given their "your data is OURS FOREVER" and "we can make derivative works form your data, or sell your data if we so wish" clauses in their T&C docs, well... (You, being a small businessman with so much to lose, did actually read those things before signing up, right???) That's why linkedin, Google and a few others never even got so much as a copy of our logo or other graphics provided to them by us (and surprisingly Farcebork did - at least they didn't claim perpertual ownership of your data!)
So the company kept single copy of its backups at the same provider?
I agree that DO should have handled the mess quicker, but well, if a company is treating its data that way, it's only a matter of time when they lose them again. And next time it might be not possible to blame DO for everything.
So the company kept single copy of its backups at the same provider?I agree that DO should have handled the mess quicker, but well, if a company is treating its data that way, it's only a matter of time when they lose them again. And next time it might be not possible to blame DO for everything.
"Our company is a leading provide of total data protection solutions. If you sign up with us, we guarantee 24x365 availability of all of your data. Your data will be kept in a secure facility and in the event of an emergency you can rest assured we will be able to get you back up and running in a very short time frame.
You do, of course, need to have a secondary computer facility ready to go should you suffer a catastrophic failure. But rest assured out vans will be on site as soon as the way is cleared. "
Concept taken from a very old BOFH possible before Mr T was even working for El Reg where the company was taking a serious look at DR setups, where backups were housed in a backup firm's storage warehouse - note that the difference basically is back then you sent backup tapes to them to house whereas today you upload your data to them. Otherwise, the process has been the same for decades - how many even well-experienced firms actually have a truly decent backup plan in place should their main datacentre fail?
........just get started with your friendly cloud provider....quick, cheap, endless capacity.....and all your computing problems are solved!!!
.....until they aren't!!
How old is this advice? -- if it seems to be too good to be true....then it probably isn't true!
A CTO of a two-person company? I only presume the other person is a CEO, then.
So, in other words, a company which has only officers and no people to do the work, which has only an office but apparently no computers and no data, raises a s-type storm on Twatter in order to boast about their Fortune 500 customers. To me this is pure unblemished virtual reality.
I hereby propose an addition to our beloved Monday's Who, me? and Friday's On call - something along the lines of "Clouded judgement" would certainly cheer me up every Wednesday!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
