back to article Furious customers tear into 123-reg after firm's mass deletion woes

The fall-out continues from hosting service 123-reg's major weekend cockup, which knocked several customers offline – with several telling us the error has effectively deleted their businesses. As already revealed, on Saturday customers' virtual servers vanished after the hosting firm ran a script containing a catastrophic …

Silver badge

Re: Too little, too late.

"Horse, meet stable door. Stable door, meet horse."

I think the problem here is that the Horse never met the Stable door. If it did, none of this would have happened.

4
0
Silver badge

Re: Too little, too late.

No horses needed. Look to the pigs instead.

0
0
Gold badge
Devil

Re: Too little, too late.

There's nothing wrong with closing the stable door after the horse has bolted. So long as there are going to be more horses along in future. At least you'd hope people might now have learned something...

So the correct quote should be: "Management, meet stable door. Bang! Ouch! Stable door, meet management. Bang! Ouch! Now, have we learned our lessons? No? OK. And again, management, meet stable door. Bang! Ouch! And for some variety, management, meet anvil! Clang! Next, pair of bricks...?

3
0

I mean seriously 123 customers.... you didn't have backups... AND trusted your provider..... ?????

Doh.....

5
3
Silver badge

I mean seriously 123 customers.... you didn't have backups... AND trusted your provider..... ?????

Doh.....

But isn't that just what all these vendors keep telling you: no need for on-premises backups, it's all in the Cloud(TM)

This is why SMB Managers keep falling for it.

16
1
Silver badge

Ive arxhived the story for when the brass decide to try and cut my budget because the cloud is cheaper.

2
1
Silver badge

In this case the customers didn't pay for a managed service. They bought a cheap service that is low resiliance and explicity not backed up, and got what they paid for.

If your brass want to go cloud then you can, and if you do it right achieve much better availability than almost anything else, do it wrong and you'll be unreliable - just like hosting yourself.

5
1

Even those who had backups - guess what? The backups got deleted too!

15 days too late for an April Fool joke!

0
0
Holmes

Elemental

Backups you say?

Nah, the cloud never fails... duh!

Now, if you'll excuse me, I have to write some more CRON jobs...

1
1

The brass won't listen, because they're more likely to hear the salespeople not the technical experts. It's been the case for decades, and will always be the case, because brass almost invariably comes from the sales side of the fence, and they only believe their own brand of lying sacks of shit.

4
0
Silver badge
Trollface

18/04 never forget!

So April 18 will henceforth be remembered as "CLOUD DELETION DAY"

0
0
Silver badge
Boffin

VM <> Cloud

What's really scary is the number of posts here that equate a VM to "the cloud".

They are not the same thing, people. If you think they are the same thing then you've missed half the point of "the cloud".

3
0
Silver badge

Re: VM <> Cloud

Please enlighten us, "Just Enough", what are we missing?

0
3

It's in the cloud

I pay money, they must be backing stuff up... did you actually check?

6
0
Silver badge

Re: It's in the cloud

Actually it's clear that backups are the users responsibility, the problem is that the backup solution provided backs up to the server itself...

I've got offsite backups, run daily so worst case I've lost 3 hours of data.. The bigger issue for me is that if I reimage and restore service they will overwrite it when they recover my original data.. And that data may be corrupted so it's better for me to wait it out and see what the manage to recover before restoring the backups.

Though restoring to a different provider is an option... I'm definitely mirroring with someone else.

5
0
Silver badge

Re: It's in the cloud

What i would do is restore your backup to a competent hosting company. Then get your dns tramsferred and backip/restore as appropriate if they ever get your data back (wishful thinking )

0
0
Silver badge

Re: It's in the cloud

Danny, more or less what's happening. :)

0
0
Silver badge

Re: It's in the cloud

Personally, I would say that it is the customer's job (for these services, low cost and unmanaged) to ensure there are reliable backups, just as it would be with self-hosted systems.

HOWEVER, 123-reg's failure here is a clear case of negligence, probably gross negligence, and I would expect a clear cut case in court. If I did this as a contractor, I would expect to be sued, whether the client had backups or not.

Also, this sort of service is often used by non-technical people. They don't understand the risks of not having a backup, or expect the service provider to back up for them.

Any professional worth his salt running anything on this kind of server will have a robust backup and DR scheme, and will have tested restoring the backup on both another system from the same SP and on alternative systems. The only people outside 123-reg I can't feel any sympathy for are those IT professionals who lost data through neglecting backups. They should hang their heads in shame. While in this case the problem came down to human error, there are numerous failure modes which are down to dumb luck (multiple disk failures, natural disasters, etc) which they should have been covering off with a decent backup & DR plan.

1
0
Facepalm

Re: It's in the cloud

*"...Any professional worth his salt running anything on this kind of server..."*

I'd question whether there was anything "professional"... or "salt-worthy" about anyone who does this stuff for a living and hosts his clients' sites on the internet equivalent of the Pound Shop.

0
1
Silver badge

Wait this is real? * could of sworn I read a similar but hoax story on el reg.

1
1

I Ran A CRON Job On My Bank!

Has anyone ever heard of Identify - Archive - Delete with the delete portion only being done when you are sure the data is no longer required, happens all the time in the simplest of environments like email programs and the ubiquitous windows Trash Can.

The result of their script should have been to mark for delete only! Where did these clowns train?

My business is in ruins today and they cannot even give me a straight answer.

Am I surprised? I shouldn't be! I was going to leave them last time they lost my dedicated server, but then I thought well at least if I put it all on a VPN then there will be backups and disk images - How wrong I was. I will not be rebuilding anything on 123-reg. I will spend my efforts and money elsewhere. I will also be seeking legal advice.

And the final cheek was that they tried to collect their Direct Debit for my VPN today. Thankfully I have recalled it, bearing in mind that the payment is monthly in advance I think I have the right, don't you?

I'll just tell them I ran a CRON job on my bank!

2
2
Anonymous Coward

Re: I Ran A CRON Job On My Bank!

Fuck, yeah, I'm been running my bank on dees clowns 123 an' now my customers said that 'is check is bounced an' 'e can't pay nobodies.

I'm gonna take my bank 'n shit some place better. Why these gangsters charging me £££ for this.

Bank of AnonCow is changing hosterz!

1
1

Re: I Ran A CRON Job On My Bank!

Better off than with HSBC!

2
0
Anonymous Coward

Re: I Ran A CRON Job On My Bank!

So you didnt think of creating offsite backups after 123 lost your dedicated server last time?

Me thinks you need to rethink your setup if its that critical!

0
0

Strange email from them today

I got an email advising that a site renewal was due

Site was probably moved 5 years ago

The date for site closure in the email was in May 2012

Have their email servers suffered a timewarp?

2
0
404
Silver badge

Re: Strange email from them today

Somebody dusted off tapes from 2012... 'Oh Hey look what I found!' lol

6
0
Silver badge

Re: Strange email from them today

Those 2012 tapes will be the back ups...

3
0
Silver badge

Re: Strange email from them today

So.etjing went seriously wrong internally with them today.. I tried to log into support 4-5 times and got a different persons details and tickets every time.

Called them to advise and they sorted it pretty quick.

Also had a brief outage because they updated their DNS today.. Apparently.

1
0
Bronze badge

Backup

I'm not sure if they offer an unlimited bandwidth vps package but if they do then you should be running an automated backup at least once a week !

I had the same thing happen to me and was backing up the entire Vps every other day. Downtime about 12 hours including phoning every customers and backing up their locally cached email that was outside of the backup window. All I lost was some server logs and sales that may have happened in the downtime. Needless to say the backup is now daily.....

2
0
Silver badge

Re: Backup

It is unlimited, I push 5-10GB of backups off various VPS every night to a FTP in the office... I've potentially lost 3 hours worth of data, so not terrible.

2
0
Gold badge
WTF?

Hmm.

.....this isn't the first time 123 has lost our data in the last few months......

Fool me once.......(!)

This is the sort of thing where you really shouldn't give second chances.

4
0
Silver badge

Soo....

A service explicitly not backed up - and the customers had no backups.

I bet they'll survive this even if some of the customers don't.

However.... the discussions here do bring back the point about why tape is great. Having your "backups" be someplace the bad guys can delete them means that the bad guys WILL delete them (This has happened on several occasions).

With major financial institutions now decreeing the cloud is good enough for backup, whose head will roll when the inevitable happens?

0
0

123-reg are not the first and not the last to have this issue!

They are not the first and they will not be the last to have something like this happen. Any one who has a business or mission critical services should have either a backup or pay for backup services. It's not rocket science you make sure that really valuable stuff is protected!

2
0

Yeah

But how sacked is that one dude - seriously?

0
0
Anonymous Coward

The Cloud...

Other people's computers you have no control over.

2
1
Silver badge

Re: The Cloud...

Plenty of companies on Twitter offering to host the websites of victims' on their "rock solid cloud" hosting.

It's all about the money, money, money.

0
0

This situation is exactly why I sync all of my data between two different hosts and with two different suppliers. I then have a load of FTP space where the VMs are archived just in case. It is a pain to set up and is not fun to pay for, but it protects me against other peoples negligence or going bust, major outages etc. You really must only trust yourself if something is so critical to your business because although it is their fault it is your problem to deal with your customers and they will be asking you why you didn't keep a backup.

0
0

Am I missing something here?

I wonder if the staff member who wrote the script still works there or if someone new has come along and tweaked it do to additional stuff and stuffed it in the process.

I've seen this before when I worked in IT support, normally the new guy checked it does the NEW functions properly but doesn't double check the old ones still work, then runs it on live servers.. then we get another new guy..

0
0
Silver badge

A bit late as commentard, but better late than never, eh?

As sysadmin I hate running timed delete jobs, I prefer to do deletes manually.

Because boo-boos can happen.

Rather let the script do backups etc, and so on, that is needed, but when it comes to deleting (or dropping databases), rather do it manually.

Because Mr Murphy loves to do on-the-fly coding as well.

2
0

You get what you pay for

Go to the 123-reg site and the first thing you see is

"Search for your perfect domain from only 99p"

That should be telling you, this is cheap and cheerful web hosting. Which it is. If your website is business critical and you build it on that without any thought to backups, this is what happens. Even my Wordpress vanity site is backed up to my home computer regularly, where I've checked that I can restore to a local XAMP server. There's nothing of real value there at all, but that sort of system is surely just a habit for anyone who has worked around computers for a while.

There's nothing wrong with having your website hosted by an outside agent (which you can call the cloud if you must). But if you've got a business that depends on the web then you need to have someone *on your staff* who understands enough to at least know what questions to ask.

0
0
Bronze badge

Re: You get what you pay for

Go to the 123-reg site and the first thing you see is

"Search for your perfect domain from only 99p"

That should be telling you, this is cheap and cheerful web hosting.

err no, that would be telling you it's cheap and cheerful domain names. You need to look at the hosting packages to determine what sort of hosting company they are...

0
0
Silver badge

Alternative low cost hosters?

I'm not a 123 customer[1] but I'd like to know who the technorati use...

[1] Vidahost.

0
0

Re: Alternative low cost hosters?

Three possible answers:

1) Hard core nerd - Home server(s) with UPS and redundant network connections

2) Professional (if my site goes down I can't pay the mortgage) - AWS or similar, spread across availability zones, tested backups, tested failover to a static site served by a CDN.

3) I'd like my site to stay up but without spending £10k plus - Use any old cheap dodgy hosting company but ensure that you have regular tested backup (stored somewhere other than the dodgy hosting company) which you can upload to somewhere new if it all goes Pete Tong.

0
0
Silver badge

Re: Alternative low cost hosters?

Personally I use Digital Ocean and look after the VPS's myself. They are then backed up to an Azure storage cloud. I have a VPS with OVH that is dormant but is configured with my clients accounts, so that if DO do a Linode and go AWOL, I can restore the sites from Azure and switch domain names for prolonged downtime.

I've never had to do this though. Thankfully!

0
0
Anonymous Coward

Sickapedia

Sickapedia is still down from a web hosting issue :(

0
0
Silver badge

Re: Sickapedia

Yeah that's been down for ages... They said recently their disks were with a data recovery company... Probably did the same thing....

0
0

"One customer claimed this practice "is frowned upon in the hosting world... I discussed the issue with a few acquaintances who have experience running servers in a commercial environment and all of them were shocked to hear that 123-reg 'automate clean-up of servers'"

Having worked in mass hosting, the only option for cleanup is automation for large scale platforms and I doubt there's a single large hoster on the planet that doesn't do it, particularly for low-cost hosting. At most large dedicated server companies, provisioning, building and deletion are also automated. There's a real risk, obviously, of mistakes happening but the real surprise is it doesn't happen more often.

1
0

What I want to know is how they managed to delete live instances.

I mean, that's basic, basic stuff.

Is this VM flagged for deletion currently in a running state? Yes. ABORT ABORT ABORT.

You can automate suspensions (after all, you just need to pause, or down the VM - pick whichever suits your flagging method to determine how agressive you want to be) but you should never, ever authorise the scripted deletion of a VM without checking if it's running first.

Hell, I'd not even want to automate the deletion of suspended servers personally (Although I'll accept that beyond a certain scale, it's unavoidable), just in case you get the syntax wrong and do a 123-reg, as I suspect that's what's happened here...

Steven R

0
0
Unhappy

Real customer hit

http://www.bbc.co.uk/news/uk-scotland-highlands-islands-36080667

2
0

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018