back to article 123-reg still hasn't restored customers' websites after mass deletion VPS snafu

Angry customers of hosting biz 123-reg, who have been left without websites after a major cockup by the firm over the weekend, are continuing to report problems. Problems emerged on Monday, after the company confessed to virtual private server customers it ran a script containing an catastrophic error at 7am on Saturday as …

  1. TonyJ

    Hmm...

    Ok, it may not be a managed service and yes, even if it was, wherever possible the customers should have had their own backups but 123 have a duty of care here...it was their script that caused the problem therefore to crow about how much they're investing in recovering the data seems a wee bit disingenuous under the circumstances.

    I would also be interested to know if there were warnings sent out that this work was going to take place and as such, that their customers should ensure they had a suitable backup just in case. Also missed sales opportunities..."Hey you have no backup...what if...? Why not pay us a little to help you here?..." etc. Of course, the cynics might say this would work to advertise exactly that.

    1. Billa Bong
      Mushroom

      Re: Hmm...

      "Whilst the 123-reg VPS product is an unmanaged service, we have committed a large number of resources to help restore services back to normal as quickly as possible"

      You're more temperate than I would be (even though I moved away from 123-reg a long time ago due to this sort of attitude). I'd be spitting nails! It's almost, but not quite, saying "you know what, guys - we can run a script and delete all your data and we don't have to do anything about it. it's all down to you to recover. But we will do something about it, because we're good guys. Here's a video of a fluffy bunny eating a carrot to amuse you, which again, we didn't have to provide but we did because we're the good guys."

      While I accept that customers not taking their own backup us a little foolish, it's bordering incompetent for a service provider to not provide service assurance through maintenance, period. They're doing the work which has risk, they should have a mitigation plan in place in case of error causing this sort of issue, else they need to make it very clear in terms of service that there is no guarantee at all of service delivery, which would turn most people with flourishing businesses away from them as a provider.

      While I'm on the subject, and since you brought up marketing, do you think in a few days 123-reg will turn around and say "ha ha, no one was impacted at all, it was all a joke to get our company talked about", just like the rm -rf guy? Would anyone actually go to a company that pretended to be incompetent in order to get their name in the news? Personally, I wouldn't...

      1. tin 2

        Re: Hmm...

        Agree entirely.

        "Whilst the 123-reg VPS product is an unmanaged service" needed to be at the very least something this idiot keeps in his head and never lets slip out. It's entirely unnecessary, yet he/123-reg seem to feel it's something that must be repeatedly pointed out as if that makes things better.

        I moved most stuff away from 123-reg for exactly this attitude also. I best move the other stuff quick.

      2. TonyJ

        Re: Hmm...

        "...You're more temperate than I would be..."

        Oh I wouldn't be, if I was one of the affected souls. I am just too cynical (experienced) to ever trust any company that hosts, well anything. Hosted myself or by third parties - if it's in any way important, it's backed up and stored off site in multiple locations.

        Nor am I in any way supporting the 123 response, hence my comment about their duty of care...you cannot run a script that trashes your customers and then claim on the one hand you're not accountable whilst at the same time claiming you're playing the nice guy by trying to recover what you destroyed.

  2. Jay 2
    Meh

    So it's looking like 123-reg didn't have suitable backups of the VPS configs (or similar) then? I can sort-of understand them not backing up the user data, but the servers that they control... that's pretty bad stuff.

    I'd like to think they'd spend some time going over the backup/restore setup once they've somehow managed to sort this current mess out.

    1. Mad Chaz

      My understanding from a previous article is the problem was with a clean up script. The script deletes "no longer used" volumes. Only this time, the script didn't delete "no longer used", just a lot of volumes.

      That is what is used as "hard disks" for virtual machines in a VPS environment. So basically, the clean up script went in and deleted user data for still active clients instead of just ones that aren't active anymore.

  3. Anonymous Coward
    Anonymous Coward

    Poor bloke. I know he fucked up, but can you imagine being this guy and having your mistake broadcast onto the internet twice a day?

    Plus companies are going to be on the lookout for anybody with "xxxx-2016 123-reg sysadmin" on their CV's for a while.

    1. TRT Silver badge

      I'm sure that they can remove that part of their work history. They're good at deleting things.

      1. Anonymous Coward
        Anonymous Coward

        (Zing!)

        But then you have to explain the gap in your employment history with the old "went to live with Tibetan Buddhist monks to find myself" story.

        1. TRT Silver badge

          Or...

          Couldn't you say you'd been on safari?

          Or been in the Merchant Navy?

          1. Bloakey1

            Re: Or...

            Well many moons ago I did the old Foreign Legion one.

            How about I went to sell bibles in Palmyra.

    2. Changethings

      Poor bloke, seriously, give the guy a google, the stories about customers experience with him, he deserves every minute of the horror that his company has caused and sounds like previously his attitude was very much I do not care and if you do sue us, btw here is the legal departments address for the summons.

      So I have not one ounce of pity for the person named constantly, I hope this highlights just how bad things have been previously with him and 123-reg

    3. e^iπ+1=0

      Poor bloke. I know he fucked up

      Can you imagine how it's going to go for him when he's negotiating his next contract renewal?

  4. Anonymous Coward
    Anonymous Coward

    This is the second time this year that my 123reg VPS has disappeared. Same symptoms as last time as well, so I think their scripts were already deleting some active instances even before this. No help from 123reg either, I had to rebuild and reinstall from my data backups.

    1. Anonymous Coward
      Anonymous Coward

      Fool me once, shame on you.

      Fool me twice. Restore from backup ... and shame me again.

    2. e^iπ+1=0

      rebuild and reinstall from my data backups

      Or possibly: rebuild and restore elsewhere, then seek refund from 123, then choose new hot spare vps.

  5. JamieL
    Holmes

    I worry about firms who are still naively waiting for a restore

    I read the article yesterday morning, and had a giggle. Then later in the day got an email from my sons' school apologising for their website being down and referring to a BBC news story about 123-reg and how it wasn't really their fault.

    Three things jumped out at me:

    1) I pay them a lot of money (yes, it's a fee-paying school and yes it's possible in this industry to earn enough for that) and am disappointed at the corner-cutting to save a few hundred quid a year

    2) That they didn't just buy hosting from somewhere else the next day, get their website designer to re-upload, change the DNS and carry on normal jogging. It's clear by now that 123-reg can't be relied upon

    3) Bad performance plus good excuse isn't the answer. They're pretty good at drilling that into the children but are still prepared to try it on with the parents

    1. Ian 55

      Re: I worry about firms who are still naively waiting for a restore

      Private schools are very good at over charging aka 'corner cutting to save a few quid a year'.

      Why else do you think they wanted the exemption to allow them to hire unqualified teachers?

      1. e^iπ+1=0

        Re: I worry about firms who are still naively waiting for a restore

        "exemption to allow them to hire unqualified teachers"

        By exemption do you mean that by not being state funded they were exempted from rules schools must satisfy in order to obtain state funding?

        1. Danny 14

          Re: I worry about firms who are still naively waiting for a restore

          Independent schools have always been able to hire unqualified teachers. This isnt new. They can hire whoever they like. Most independant schools are borderline broke too hence why their salaries are often less than state school equivalents.

          The bonus is not having to nail things down/lock in steel boxes etc.

      2. Vic

        Re: I worry about firms who are still naively waiting for a restore

        Why else do you think they wanted the exemption to allow them to hire unqualified teachers?

        "Unqualified" means neither "low quality" nor "low-paid".

        By being allowed to hire whomever they like, they can recruit specialists in any field. The state sector generally[1] needs at least PGCE post-graduate training, meaning the choice is narrower.

        Vic.

        [1] There are ways of getting someone without a PGCE into the classroom - but that is usually seen as "on-the-job training", and earns even less ...

    2. wolfetone Silver badge

      Re: I worry about firms who are still naively waiting for a restore

      "I pay them a lot of money (yes, it's a fee-paying school and yes it's possible in this industry to earn enough for that)"

      Can you be my daddy? I need an Xbox.

  6. wolfetone Silver badge

    Cowboys Ted. They're A Bunch Of Cowboys.

    It's interesting reading all of this unfold on Twitter. I've helped two or three people out getting some content back from their website, but not everything. But it's scary to see all of these web hosting companies offering free shared hosting to those affected by 123 Reg, and people taking them up on the offer.

    It's like they haven't worked out that it wasn't the cost or the quality of the hosting that was the problem, it was the lack of a back up that screwed the customers over. So what does the customer do? Decide "F U 123 Reg imma going to Cheapo Web Hosting, dey give me hosting fo free becuz urs is shit".

    Still, fools and their money eh?

    1. TRT Silver badge

      Re: Cowboys Ted. They're A Bunch Of Cowboys.

      Shoddy, shoddy, shoddy.

    2. d3vy

      Re: Cowboys Ted. They're A Bunch Of Cowboys.

      I've offered to host holding pages or simple site explaining why their main sites are down... Just until they get their main sites back up.

      1. wolfetone Silver badge

        Re: Cowboys Ted. They're A Bunch Of Cowboys.

        "I've offered to host holding pages or simple site explaining why their main sites are down... Just until they get their main sites back up."

        Same as me. My servers are backed up to Azure and to a separate VPS held by another provider to who I use so if my main provider screws up I can switch it fairly quickly. But people don't like paying for that safety.

  7. lpcollier

    Backups may not be the answer

    There might be another issue at play here. Heart Internet, 123-Reg's more upmarket sister company, provides backups for their VPS service. It's a separately hosted file store that's mounted via NFS, ONLY accessible from the IP address of the relevant VPS host. If 123-Reg have the same system, then it wouldn't be possible to pull backups until the VPS image is back up and running.

    1. d3vy

      Re: Backups may not be the answer

      123 do similar but from what I see they removed the restriction on IP address pretty quickly after the issues started on Saturday.

  8. wide_awake

    Just to be clear on what’s going on here (my company is directly affected).

    We have lost no data. Zero, but we're still offline. Everything was backed up off-site. What we need is a new VPS with nothing recovered or restored so that we can restore our own back up. It should take 123 15 minutes to spin a new machine up but they can't because they've lost much more than they are letting on. Unfortunately for us, our back-up will not restore on a different providers server, so we are re-building from scratch and due to finish on Friday. Ok, we learned that off site back-ups isn't enough, and of course we've learned that our business needs cannot be satisfied by 123. A hard lesson. The most sickening thing is we realised 2 weeks ago that 123 was going wobbly when we couldn't manage our DNS for 24 hours. We already started a migration plan and we were just to late :'(

    1. Chloe Cresswell Silver badge

      We lost a client's legacy site on a VPS we inherited from previous IT company.

      It came back up today.

      Unfortunately for 123, we had a replacement tempoary site spun for the client by the end of Sunday, online after talking to them on Monday, and the client is finally going into meetings for a replacement solution next thursday after 2 years of "we'll do this soon". So the VPS's site will never be online again even with 123's restore.

      For us, it's been the kick the client needed to retire the pile of asp that no one understood that ran the old site.

      1. Daniel B.
        Boffin

        Gah!

        For us, it's been the kick the client needed to retire the pile of asp that no one understood that ran the old site.

        Let me guess, pre-.NET ASP? Horrible! VB can't seem to die fast enough.

        1. Chloe Cresswell Silver badge

          Re: Gah!

          Bingo

    2. JamieL
      Alert

      ...which is why it's a good idea for your Business Continuity Plan to involve as much diversity as possible: different suppliers, different infrastructure. And of course it counts for nothing unless you've tested it, and testing means more than "just spin it up" for a quick "hello world". It means use it. For at least one business cycle (even if just on a sub-set of your business e.g. 5% of customers).

      Have just been having that discussion with my current employer: explaining that the scenario test we did on a Friday morning between 0800-0850 may tick some boxes but is not comprehensive and gives no guarantee that it'll still be working 5-10 days later.

    3. d3vy

      I provisioned a new machine yesterday... Took almost 12 hours to be set up... There must be a high load on the hosts at the moment.

    4. Vince

      The lesson that should be learned there is that a backup that is not tested is no backup at all.

  9. Domquark

    Ahhh

    The joys of the Cloud! All your eggs in one basket......

    1. Mark 85

      Re: Ahhh

      That can't be... it's the cloud.. it's forever.... Oh wait...

    2. d3vy

      Re: Ahhh

      This isn't cloud hosting...

  10. Alan 19

    Insurance

    All those who say their website is critical to their business should have Cyber & Data Risks insurance then, surely.

  11. Anonymous Coward
    Anonymous Coward

    This is what happens

    This is what happens when a company employs graduates....

    ....of a circus school.

  12. Peter X

    How many servers? How many websites?

    The article says "114,933 servers out of 115,000".... so am I correct in thinking that that's considerably more than "a minority of customers" or whatever 123-reg's original assessment of impact was?

  13. akaracing

    BAD NEWS

    This is the email we just received from Richard Winslow at 14:06 today

    Over the past few days, 123 Reg has been working to restore your VPS service following the outage on 16th April, 2016. We understand how business critical a website is to you and we have been doing everything in our power since the fault occurred to get you back online. We would like to extend our apologies to you for this.

    Regrettably, we have been unable to restore your VPS.

    We have brought in Kroll Ontrack, the world’s leading data recovery experts who will continue to work with our own experts to conduct forensic analysis on our servers to continue to try and recover data. However, at this point we strongly advise you to work on other methods to restore your VPS.

    As VPS’s are self-managed platforms, we needed to check each server individually to give clear updates to you which has taken longer than expected. Thank you for your continued patience.

    We fully understand that this is the worst possible outcome. In order to help speed the process of getting you back online, we strongly advise that you attempt to restore from personal backups. Please note that if you attempt restore from your own backups, this will not overwrite our continued efforts or impact the data recovery process.

    If you need any further information on how to rebuild your VPS, please follow the instructions here: https://www.123-reg.co.uk/support/answers/How-to-rebuild-your-123-reg-VPS-5130/

    We would like to offer you 6 months free VPS service with a year’s backup for your server from the world’s number 1 backup software provider - Acronis. In the meantime, if you need to put a holding page live you can activate your free Website Builder through your 123 Reg control panel.

    123 Reg fully understands the importance of your online presence and we have, and continue to do everything possible to recover your VPS.

    I would like to extend my sincerest apologies to you.

    Richard Winslow,

    123 Reg Brand Director

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like