UK web host 123-Reg goes TITSUP, customer servers evaporate UK hosting and domains provider 123-Reg has been struck by a weekend TITSUP (Total Inability To Support Usual Performance) that knocked an unspecified number of VPS (virtual private server) customers offline. The company posted a status message saying that the unspecified issues arose on April 16. 123-reg customer, software …
Well that sounds very professional. We screwed up and we're very sorry that you don't have a backup.
That's not even a Cloud issue, it's a hosting one, but here we go again with the consequences of trusting your data to someone else's infrastructure. Somebody goofed, which happens, and websites were deleted. Okay, human error, statistically unavoidable, but compounded by a total lack of data backup - leaving customers up the creek to provide their own paddle.
Well it's a choice and it was probably in the T&Cs, but I'll wager there are a bunch of websiteless companies that are, at this point, seriously re-examining their level of acceptable hosting costs and guarantees while looking over other hosting companies' offerings.
I know I would be.
"Well that sounds very professional. We screwed up and we're very sorry that you don't have a backup."
INNmaster - who expressed the very sympathy you are criticising - are a 123Reg customer. As such, I don't think they're responsible for making backups of other 123Reg customers' websites.
Absolutely! What a numpty I've been. I was a 123 reg dedicated server customer for a number of years without incident. Then one day they erased my server by mistake thinking it was someone else's. I had no recourse and it was my fault for not having a backup; after all I was just a customer; what rights could I possibly have claim to?
So I decided to leave, but in the interim I downgraded to a VPS thinking that at least on a VPS there would be a disc image on the cluster so total loss "JUST COULD NOT HAPPEN".
Well, time went by and I didn't get round to moving everything to a new hosting provider.
You'll never guess what happened to me at 07:21 am 16/04/2016?
What is that supposed to mean? My VPS went down at approx. 9am Saturday morning. In 48 hrs there have been 5 status updates. Most providing little information, and what they did provide being so vague as to be useless. For example they currently claim to be restoring VPS images, but won't say how quickly, or what percentage are likely to be restorable, or what criteria are likely affect the chances of your being a recoverable image.
Things break, it happens, but the communication has been the real failure.
I'm not defending 123-REG (and I'd never defend the customers daft enough to take service with these budget providers for anything vaguely important either)... but out of interest...
""won't say how quickly?"
What are you asking for? The number of KB/MB/GB a second being restored or the number of machines vs total or ???
"what percentage are likely to be restorable"
I haven't checked, but assuming backup is part of the agreement anything less than "all of it" means they've been in breach of contract. However, in terms of knowing this, this brings me to a pet niggle.
When something breaks, the first questions you get are:
"How long will it take to be fixed"
"What is wrong?"
"How long will it take"
"So when do you think it will be working again"
"Why can't you fix it sooner"
...essentially all silly questions. Normally the focus is on restoring service. Root Cause analysis is not going to happen at this stage, and often you won't know when something first goes wrong, you won't know thus what is wrong, and you won't know how long it will take either.
That said, the updates 123-REG have given are essentially just vague probably because they're a bit embarrassed, a little bit of PR management and a dash of disinterest.
What do you expect for "starting from £9.99 a month?
What do you expect from something starting from 9.99 a month?
Well, for a start for them NOT to delete EVERYTHING and maybe backups.. Unfortunately they have failed on both counts.
They don't back up the virtual machines.
And many of the affected customers choose the budget option for a reason.. They are small companies with small margins keeping expenses down is important.
"Well, for a start for them NOT to delete EVERYTHING and maybe backups.. Unfortunately they have failed on both counts."
Does that agreement say they WILL back it up (I don't know, I'm not a customer). If it doesn't, then there's no point moaning, that's your problem.
"They don't back up the virtual machines"
See above.
"And many of the affected customers choose the budget option for a reason.. They are small companies with small margins keeping expenses down is important"
Sure, but then that's part of the calculation you make - there's a place for budget providers, and indeed budget customers, but normally, from experience, budget customers expect premium service, premium resilience and premium support. It's rarely possible to get incredibly low pricing and all of those.
To be fair they do say that backups are not included... But then they provide backup software which backs up to the VPS itself... Great of you delete the off file.. Not so great if the whole server is gone.
If it was a physical server this is the equivalent of them in racking it, putting it in a skip and setting it on fire.
Regardless of whether they provide backups or not, you don't expect that!
PS. Just spoke to support, if I reimage my server and set it all up again from backups they will overwrite it when they manage to recover the VPS image (I gather this is recovery of deleted data rather than restore from a backup)
This means that even after I get my e-commerce site back up and transactional again at some point in the (possibly) near future, they will replace it with data from Saturday morning so I'll lose any data generate after that on my restored server...
And as anyone who has had the misfortune of being involved with undeleting deleted data knows, you are going to have some bit rot in there. Some percentage of your data will have odd corruptions in it; better hope it's not in a key part of your database or your will end up with a 'restored' service that won't actually run because some file somewhere has a 1 where it should have a 0.
So even if you do restore from your own backup, they could end up overwriting it with something broken anyway.
ಠ_ಠ
More importantly, do they have enough free unused servers to ensure that anyone restoring themselves isn't going to overwrite somebody's deleted data? Any disk that has had data deleted from it needs to be taken offline completely and mounted read-only otherwise things will only get worse.
"To be fair they do say that backups are not included... But then they provide backup software which backs up to the VPS itself... Great of you delete the off file.. Not so great if the whole server is gone."
My reading of not providing backups is that if I mash up my files from day A to day B then I can't restore back to A but I would expect at the server level some form of snapshot/backup, not a restore to a completely empty box.
Glad I'm not with them :p
To which the answer is "It depends how long I have to spend answering your questions".
No it isn't - the correct answer is I'll publish a one sentence status update every hour or two so I don't keep getting bugged by people asking how it's going.
This is what management are for, it's not like they can sweep this issue under the carpet, so own it.
If I leave 123 Reg it won't be because of the failure - all platforms fail eventually. It will be because I wasn't kept informed enough to keep my customers informed.
Trouble is, I've never come across a cloud/hosting/service provider that did keep people informed.
"Trouble is, I've never come across a cloud/hosting/service provider that did keep people informed."
My current provider has had a couple of issues over the past few years.
There was even an IP range change (having to give back small slices in order to get larger slices), there have been data centre level DDOS and a router screwup.
But I have always had an email from the main technical support within minutes of it kicking off (I was actively online for one of the events, and I hadn't finished bashing out my "what's up" email before theirs hit my inbox).
Regular updates of what they are trying along the way, then a decent breakdown of the issue after.
Makes life so much easier if people admit that things go wrong, and deal with their customers as if they could do it as well...
Who? corgi tech
I was a fairly early customer and agreed to switch data centres for bonus features (i.e. I pay a lot less than I should) - but it's still pretty good value anyway. I don't use it for anything particularly critical, but I really notice when it isn't there...
I put these bozos in the same bracket as Farthosts. I have migrated many customers from both providers over the years. Not so much for reasons of reliability (although both have been shaky at times) but the customer service was always dreadful and lack of technical control.
I definitely understand choosing cheap providers because of cost - if you won't die over a day or so of downtime then thats a perfectly valid decision to make. But if downtime is going to cost your business serious money, it's probably better to not be using budget mass-market web hosts where you are one account of 10,000 and instead have a managed service, or better still run from multiple providers. Compare the loss due to downtime vs the cost of improving availability.
One thing I would say is separate out your DNS hosting, domain registration and web hosting. Easier to juggle things around in times of brownout.
But I can't help wishing them the worst.
Many moons ago, I used to teach a web design module at a college. Whenever I got to the lesson on registering domain names and web hosting [with accompanying "who to avoid like the plague" caveats], there would always be a few students who'd proudly announce they'd already registered a domain name and bought web space, in advance of the lesson.
My heart would sink because, 99% of the time, said domain name would have been registered with 123-Reg, or similar "Get a Domain Name and 25 Years Hosting for Only 99p!!!!!" Mickey Mouse operation and I knew I'd be spending most of my break and end-of-lesson time wrestling with their fucktarded Fisher Price "control panel", trying to configure DNS servers or <shudder!> upload a website via a browser based form.
I know this is not about 123-reg's business practices, nor do I agree with the criticism of their technical support - I have always found them excellent when you can get through (which of course means when there's NOT a big outage), but there's no harm in reminding of this.....
http://www.theregister.co.uk/2014/04/03/123reg_shrugs_at_complaints_over_sudden_domain_transfer_fees/
While I agree (I've been with them for 5-6 years with no issues) normally their support are great.. In this instance though they seem totally unprepared.
I'm going to stick with them in the hope that they have learned from this.. I certinally have and will have mirrors of my servers somewhere else too.
Email from 123-reg.....
I am writing to you to explain what happened to some VPS services on 16.04.16. This email is to detail what our steps have been. I am committed to open communication with all customers and would like to take this opportunity to explain in detail.
So what happened to some services? As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore. What have we done? We have been working with an extended team of experts and have left no stone unturned. Our teams have been working long into the night to restore as much as we possibly can. We have also invested in external consultants to recover, in the best way possible.
We have recovery running on the VPS servers and some are restoring to new disks. We have also begun copying recovered VPS images to new hosts and we expect some VPS to be back up and running throughout the night and in to tomorrow.
So, a bug in their script reported some customers as having no VMs running when they actually did? And the "cleanup" job was to simply nuke the apparently spun-down VMs that weren't actually spun-down but were live?
Shoddy stuff. VERY dangerous thing to automate IMO, they should have just used the script to generate a report and take it from there.
The problem is that no-one will tell you if they have a brilliant hosting company, because they don't want that company to acquire so much new business that they become over-stretched and flaky. I did a decade of trial-and-error (including an early flirtation with 123) before I found my lasting life-partner - and you're not getting their number from me!
They're all consumer-level providers - fine for little Jimmy's scout group, but that's about it.
If you're looking at them, then you probably don't have budget for a proper supplier with a dedicated account manager and SLAs.
You could always do it yourself - but then you'd need to fork out for a pipe with an SLA ('cos you won't want to use your consumer-grade ADSL) plus a DR site... Which you'd already be doing if you could afford it.
Domain Reg: Namecheap
Hosting [VPS]: Linode*
*I know Linode had their own 'Keystone Kops' period a few months back, and I was first in the queue to stick the boot in, in these comments. But, weighing everything up, I've had 1 day downtime and about a week of crossed fingers, in almost seven years of running a couple of VPS with them. So, in the cold light of day, that's good enough for me.
i resell for these guys. they host my stuff and about 40 of my business customers too
shameless affiliate link below, you can go direct if you want, im sure you can work out the url, but Gods honest truth these guys are golden.
http://billing.host100.co.uk/billing/aff.php?aff=004
medium sized outfit, superb service, uk based datacentres, uk based support engineers - they all know what they are talking about, they look after the servers. call them up and you are right through to someone who knows what they are talking about, what you are talking about and how to fix the issue.
DigitalOcean. They are utterly brilliant. They now have a datacenter in the UK as well (London). The big boys (Azure, AWS and Google) also do good small VPSes but they will be more expensive. Linode is excellent as well but they don't do a 512 MB VPS at $5/mo like DO.
Pretty bad but not as bad as hearts balls up...
And people wonder why I'm paranoid about hosting solutions.
(for the record I'm about to start on migrating from a server 2008 to 2012 dedicated box. At this point with all the recent vps boxes issues, I still see no reason not to stick to dedicated hardware)
"Pretty bad but not as bad as hearts balls up..."
Heart has been a part of the same group since 2011.
Mind you, if you're referring to Heart's power outage that happened a couple of months back- corrupting our MySQL installation in the progress- that was bad, but I'd have said a complete loss of multiple VPSs was worse still.
I have had a very long period of reliable operation of a domain name registration with email forwarding, over fifteen years.
This isn't part of this cock-up, for which I am very grateful. My email still works. I have been receiving email forwarded through my domain.
This is bad enough. If they mess up the Domain Name registration side, they're going to be dead.
And what happens if they do drop their Doman Name database in the shit? Who will know who has a right to use a name. Tesco, to pick an example, have trademarks registered, and lawyers. It's easy to prove they own it. What proof do I have?
And whilst you are looking at SLA's ask them for their SLA on their DNS service. When they suffered a full weekend outage a few years back the response I got was it was free and therefore subject to a best efforts policy. Needless to say I moved every domain elsewhere very quickly and have not touched them since.
These guys must host their own email. They want you to email for updates but their email address doesn't work:
Delivery to the following recipient failed permanently:
vps-team-123@123-reg.co.uk
Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain 123-reg.co.uk by edge01.webfusion.com. [92.60.112.26].
The error that the other server returned was:
500 Message rejected
Probably overloaded! I sent one a couple of hours ago and so far it's not been bounced back, but hey, they've probably run a script to process email and deleted it :) (Smiley face as I'm trying to stay positive over this shambles)
NOTE: Still not had an update on my server tho....
not one has been without billing issues. Sometimes even in my favour! At least I configure them securely enough that one hosting company's admins had trouble getting in to test things when I reported the clock being wrong enough to cause problems (the host box's fault, and seriously who doesn't use NTP).
If an outage like this happened to me (as it has more than once) I would immediately start restoring my backups to a completely new provider, because you don't know if or when the original provider will fully recover, because restoring and DNS propagation take a while, and because better providers almost certainly exist but you have to test them.
I currently use a VPS at transip.eu in Amsterdam and they have been fairly reliable.
Hi guys. I feel sorry for the 123reg victims including myself. Decided to leave them right away, whatever they say. Am now with the company Linux Hosts Ltd. They claim to have the lowest customer account cancellation rate. Many different VPS plans from £10.43. So far so good!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
