"doing everything in our power "
Except getting the chequebook out for compo
They should at least refund the previous year to those customers at the bare miminum.
Customers still unable to access their websites following a mega cock-up at hosting site 123-reg over the weekend have been offered six months' free VPS and backup recovery services as a sweetener. The biz 'fessed up to customers this week that a script containing a catastrophic error which was run on Saturday (16 April) took …
This post has been deleted by its author
So, 123 lost the web sites contents - their fault for not testing things properly
The customers haven't got backups - customers fault for not doing backup in the first place.
Then the customer get free hosting for 6 months.
Er, so what will they be hosting then some form of "This customer has not created a site" holding page ?
I wonder if anyone who has only static files has tried to recover from archives such as "the way back machine" or similar archives. See https://archive.org/web/
If you were in the sorry situation of having no backups of your own, then way-back-machine or google-cache might be your best option. EXCEPT way-back-machine won't show anything if the robots.txt file is no longer being served (or indeed, if it's set to block bots).
Anyone know what 123-reg have been serving for sites that have been nuked? It would be interesting to see if they do serve a robots.txt file.
Yeah, NOW you do.
But you didn't take it seriously enough BEFORE the brown stuff got widely dispersed, did you ?
And the irony of it all. 67 out of 115000 servers. A 0.0006% mishap that is positively trashing their reputation. I think other hosting services would do well to sit up and take notice. Operating at minimum financial cost is something that can cause great financial cost down the line.
Operating at minimum financial cost is something that can cause great financial cost down the line.
Unfortunately this argument rarely sways beancounters unless something tragic happens to the Finance server in the fortnight before the final budget meeting...
Does anyone really believe the 67/115,000 number? Someone suggested that they were cherrypicking numbers - 67 physical machines, out of the 115,000 virtual machines, which would probably be a few percent of their business (how many VMs on a server?).
...was another bunch of customers passing the sound barrier to get away from you as fast as possible.
If I had been a customer affected by this cock-up, I wouldn't be wringing my hands and whining about compensation, I'd be looking for someone more competent and moving everything away at the first opportunity.
Six months of FREE!!! crap is still six months of crap, and probably comes with strings attached. No thank you.
While I understand the anger, I'm not sure it does any good to assign blame for this.
I am not familiar with 123-reg, but they sound like a low-cost operation. As long as they were transparent about what they sold, and delivered what was sold, they're an honest business. Backups aren't free -- if they were, then every customer would have their own backups anyway, right? Yes, they screwed up and should fix their mistake, but mistakes do happen, and it's not clear that this was more than a very unfortunate mistake.
Likewise, customers often don't understand the relevant differences between different products. I see this a lot in consulting. Someone can hire a larger firm for a lower hourly rate to do the same work. If that's all you see, the choice is pretty simple. Even if they have a comparison list of the differences in what they're buying, that's usually in some form that makes sense to the seller, not the customer. I don't like to pay extra for "magic beans" either.
Yes, this sucks. Yes, it makes 123-reg look unreliable. Yes, it makes some of their customers look naïve. Learn how to prevent this in the future. That responsibility falls on all parties, not just 123-reg.
Oh, and I do plan to use this as an example of what can go wrong when you don't understand a product.
You could nearly mount that argument if the failure was caused by a tsunami hitting their data centre. The script was run by them for them with no customer benefit. They did it in a prod environment without any fallback plan and without giving notice to their customers. Inadequate precautions were taken. Blame is the right response here.
Adam, I completely agree that 123-reg could be blamed. I also think that customers could be blamed. My point is that blame doesn't help.
"The script was run by them for them with no customer benefit." Do you suggest that hosting companies should not do this?
"in a production environment without any fallback plan" I suspect most companies run scripts in prod. The lack of a sufficient fallback plan was indeed a serious mistake.
"without giving notice to their customers" Do you suggest that hosting companies should notify their customers any time a script is run that touches their service?
"inadequate precautions were taken." Hindsight is wonderful. Is your point that in the future, be sure that all precautions are adequate?
"Blame is the right response here." It's a valid response. I just don't see how it improves things for the future.
I used to work with a large financial services company that was known for blaming people (and firing them.) It made the staff wonderfully careful, right up the point where things began to break. At that critical point, the clever folks ran like hell, leaving the less gifted people, junior staff, and contractors to solve the problem while dealing with managers on a witch-hunt. It was in nobody's interest to understand the real causes of a problem (either you were safe, or already fired.)
I'll address this, as a senior lead in a very large global professional services firm. (Anon for reasons)
"The script was run by them for them with no customer benefit." Do you suggest that hosting companies should not do this?
"in a production environment without any fallback plan" I suspect most companies run scripts in prod. The lack of a sufficient fallback plan was indeed a serious mistake.
"without giving notice to their customers" Do you suggest that hosting companies should notify their customers any time a script is run that touches their service?
"inadequate precautions were taken." Hindsight is wonderful. Is your point that in the future, be sure that all precautions are adequate?
These three points are simple, a proper change control process should cover changes within your internal environment and also handle vendor work. We receive multiple updates per day from vendors in changes they are making within their environments that may impact us, be it internal DNS changes, VM maintenance, mailbox maintenance. This is all built into the contractual terms and makes it a hell of a lot easier to unpick things when something does go wrong.
CLEARLY - given that this is the first such catastrophic fuck-up in 123-Reg's history that I'm aware of (and I've been a customer for about ten years) - they've been getting things right so far.
I don't believe FOR A SECOND that if they were really routinely recklessly fucking about in the live environment that a similar SNAFU wouldn't have happened years ago.