back to article Furious customers tear into 123-reg after firm's mass deletion woes

The fall-out continues from hosting service 123-reg's major weekend cockup, which knocked several customers offline – with several telling us the error has effectively deleted their businesses. As already revealed, on Saturday customers' virtual servers vanished after the hosting firm ran a script containing a catastrophic …

  1. wolfetone Silver badge
    WTF?

    Someone has lost their website, you don't have the 50GB back up file to restore their website, but it's alright because you've fixed the cron job so that deleting servers doesn't happen again? And that last bit is meant to restore faith in your company?

    OH YOU'RE DIDO HARDING IN DISGUISE!

    OH YES YOU ARE!

    HARDING IN DISGUISE!

    YOU UTTER CLOWN!*

    *To be sung along to the tune of "Devil in Disguise", by Mr.Elvis Presley.

    1. Havin_it
      Childcatcher

      For some reason I thought of Stewart Lee's delivery on reading that, not the redoubtable Mr. Presley.

      [Icon: Thought I'd strayed onto the Grauniad by mistake]

      1. Captain DaFt

        "For some reason I thought of Stewart Lee's delivery on reading that, not the redoubtable Mr. Presley."

        I read it with John Cleese's voice as Basil Fawlty. Much more hilarious.

    2. Anonymous Coward
      Anonymous Coward

      Not at all comparable to the TT incident which was pretty trivial. Rather more serious than employee hacking allowing access to the same level of bank detail that is available when people pay mail order by cheque, which is all that the TT issue amounted to..

  2. Anonymous Coward
    Anonymous Coward

    I bailed on these jokers 7+ years ago, when they had a massive DNS outage (You had ONE job!).

    Since then, many reasonably priced and technically competent registrars have entered the market. And they have decent control panels not designed by the CEOs nephew.

    1. HmmmYes

      Did he use MS Access?

      All CEOs nephews use MS Access!

      1. Triggerfish

        Frontpage.

  3. Stoneshop
    Devil

    "Hosting biz still working on fix for outrageous outage"

    "Hosing biz" would be rather more apt.

    1. Darren Sandford

      Re: "Hosting biz still working on fix for outrageous outage"

      Hosting blitz?

      1. Paul

        Re: "Hosting biz still working on fix for outrageous outage"

        Hosing Blitz?

  4. This post has been deleted by its author

    1. Stoneshop
      Holmes

      Re: Takes courage

      To run rm -rf with parameters in a script.

      First, you test for the parameters to have valid values

      if [[ -z "$1" || -z "$2" ]]

      then

      echo "Variables not set"

      else

      echo "rm -rfi $1/$2"

      fi

      as well as turning whatever command you're trying into an echo statement. Similarly, you test for reasonable values retrieved from the database.

      Then, you run this on a sacrificial environment.

      Then, you run this on a sacrificial environment, capturing the output. You inspect this to contain the commands you expect.

      Then, you run the captured output as a script. After you've verified that it does what you intended for the first couple of lines, you ctrl-c the script, remove the '-i' and re-run it.

      Then, you verify the restorability of your backups.

      Then, you verify the restorability of your backups again.

      Only then you run the script on your live environment.

      .

      1. pdh

        Re: Takes courage

        You can't really trust those test -z's... if $1 and $2 are space characters, or slashes, or periods, (and I bet there are more), you still lose. And you probably can't afford to perform that careful testing procedure for every new set of $1 and $2 values.

        Writing "rm -rf $variable" does require a certain hubris.

        1. AndrueC Silver badge
          Stop

          Re: Takes courage

          I'm a software developer and speaking frankly when I have to code a deletion against any persistent storage it always give me the heebie jeebies. I'd always far, far rather have an 'I Am Active' flag that I can just clear.

          I've been coding for thirty years and spent fifteen of those as a data recovery engineer and writing DR software. These things are what experience is made of.

        2. Stoneshop

          Re: Takes courage

          And you probably can't afford to perform that careful testing procedure for every new set of $1 and $2 values.

          That's why you write the resulting commands into a file, which you inspect before running it. I also mentioned checking the results from the database for sane values

          And if you can't afford testing potentially disastrous stuff like this, you shouldn't be sitting at a keyboard from which you can issue such commands.

          1. Doctor Syntax Silver badge

            Re: Takes courage

            "That's why you write the resulting commands into a file, which you inspect before running it. I also mentioned checking the results from the database for sane values"

            And take the path variable that you've constructed and feed it to ls, just to see if what you expect happens.

        3. Anonymous C0ward

          Re: Takes courage

          In my Linux tinkering I see a lot of tests like that stick a letter x in front of the variable, so the comparison is never looking at a blank.

          1. Paul

            Re: Takes courage

            I've seen that. Such bash horrors are written by people who don't understand how to use single and double quotes properly.

        4. Anonymous Coward
          Anonymous Coward

          Re: Takes courage

          Those -z's actually work fine if you have quoted the variables (as the OP did). It breaks if you forget the quotes though - which lots of people do. It makes me cringe every time I see this.

          Shell scripts are a rubbish environment for writing any sort of robust task. You can get slightly saner behaviour with:

          #!/bin/sh -ue

          Then:

          * Your script will abort if you try to use an unset variable in an expansion

          * Your script will abort if one of the commands it runs gives a non-zero return code which you don't test for

          But at the end of the day, you would be *far* better to use a real programming language.

          1. fnj

            Re: Takes courage

            Watch out, using sh -e or set -e is usually crap. It is not that unusual to execute commands in the shell which are expected to return non zero in the ordinary course of events; e.g.:

            #!/bin/sh

            set -e

            x=`expr 0 \* 1`

            echo "$x"

            The echo command is never executed, but there is no diagnostic. That'll give you a bad day debugging. It's the same behavior in POSIX sh and bash.

            You're right about sh -u or set -u, though. It's a lot easier than testing every use of a variable with [ -z "$var" ], and does give you a built-in diagnostic when it aborts. As others noted, neither one is a complete protection, of course.

      2. Anonymous Coward
        Anonymous Coward

        Re: Takes courage

        $1 and $2 referring to how much people paid for the service, I presume ? Let's face it - 123 is hardly a 'premium' establishment and and company big or small not backing up their own files (Whether entrusted to 123, MS, AWS or a cowboy down the road) is not doing it right.

        1. Danny 14

          Re: Takes courage

          Plus, why the fuck should a db/vm monitoring script NEED to live delete? Surely it would be better to at least FLAG these to a human? Afterall, it seemed the script had the power to do some major deletions.

          Idiots.

      3. raving angry loony

        Re: Takes courage

        -z with double-quotes around the variable? What happens if the variable is "space" or "tab"? Ah, right, same level of fuckup.

        Get a new job son, you're terrible at this one.

        1. Stoneshop
          FAIL

          Re: Takes courage

          Get a new job son, you're terrible at this one.

          Did I say it was the only test? Did I say that the test passing would unconditionally execute rm -rf $whatever immediately?

          Better get a new reading comprehension module son, it's broken.

          1. Destroy All Monsters Silver badge
            Holmes

            Re: Takes courage

            Code this shit explicitly.

            Using a programming language.

            Any programming language.

            Not murderhack "how is this interpreted NOW?" bash.

            1. fnj

              Re: Takes courage

              @Destroy All Monsters:

              Code this shit explicitly.

              Using a programming language.

              Any programming language.

              Not murderhack "how is this interpreted NOW?" bash.

              Oh come now. This class of error has absolutely zero to do with using bash specifically, or even using a shell based on POSIX sh. It has everything to do with the hazards of constructing arguments dynamically, no matter what the language. MySQL, python, C, go, or anything else: they all entail the same hazard.

            2. Stoneshop
              Coat

              Re: Takes courage

              Any programming language.

              APL? Brainfuck? Forth? Mumps?

  5. XSV1

    M-Web

    M-Web Business in South Africa had a similar cockup with their VMWare servers last year. They lost EVERYTHING on the affected servers.They couldn't even roll back because the backups were deleted as well.

    1. Doctor Syntax Silver badge

      Re: M-Web

      "They couldn't even roll back because the backups were deleted as well."

      It sounds as if they didn't have backups, just online copies. A backup is on removable media. In a firesafe. In a different building. On a different site.

      1. Danny 14

        Re: M-Web

        And then the backups are archived to another set (because im paranoid)

      2. Stoneshop

        Re: M-Web

        A backup is on removable media. In a firesafe. In a different building. On a different site.

        And has multiple generations.

    2. FuzzyWuzzys
      Facepalm

      Re: M-Web

      As a wise man once said, "It's not a backup until it's restored.".

      Test, test, test and test again!

    3. Adam 1

      Re: M-Web

      I'm not sure that backups on their own would save the day here. It's one thing to have the said offline tapes. It's quite another thing to be able to restore many hundreds of TB in anything approaching "reasonable time".

      1. werdsmith Silver badge

        Re: M-Web

        I thought that users of ISPs for website hosting had responsibility for their own backups.

        I know if mine were wiped now I could restore it from my own backups (less the updates by users that have happened since 4AM) in the time it takes to push up the wire.

        1. XSV1

          Re: M-Web

          "Normally" users are only responsible for backups of their data, not for the image of their environment.

    4. Anonymous South African Coward Bronze badge

      Re: M-Web

      He he he he, I can remember the outcry on http://mybroadband.co.za/vb/ when that happened!

      Was not pretty!

      Wonder how many clients jumped ship right there and then.

  6. joshimitsu

    Maybe it's a hoax

    On a much larger scale than what this guy did:

    http://meta.serverfault.com/questions/8696/what-to-do-with-the-rm-rf-hoax-question

    Original post where he pretends to delete everything:

    http://serverfault.com/questions/587102/monday-morning-mistake-sudo-rm-rf-no-preserve-root

    Is 123-reg a "you gets what you pays for" company? (i.e. rock-bottom)

    1. Destroy All Monsters Silver badge
      Trollface

      Re: Maybe it's a hoax

      Truth is stranger than stackoverflow.

      (Synchronicity on the Internet? Sysops everywhere, beware!)

    2. Anonymous Coward
      Anonymous Coward

      Re: Maybe it's a hoax

      Have you thought that that post may not have been a hoax and the guy who posted it was the guy who had made this cock-up???

  7. h4rm0ny
    WTF?

    I can't believe this.

    I accept that the company has actually issued these statements and I accept that El Reg is reporting it. But I honestly find it no easier to believe than if I saw a cow begin to levitate in front of me.

    A company could just not be this incompetent. It just doesn't compute in my brain.

    1. Anonymous Coward
      Anonymous Coward

      Re: I can't believe this.

      A company could just not be this incompetent. It just doesn't compute in my brain.

      Oh it does compute in my brain. But maybe that's because I believe that dark energy and stupidity are one and the same. The very force that appears least organised and most chaotic, most elusive is yet the most prevalent in the universe, and the very thing that holds it altogether. I think a small white cartoon dog beat me to this important theorem, though:

      http://dilbert.com/strip/1996-07-21

    2. Anonymous Coward
      Anonymous Coward

      Re: I can't believe this.

      "A company could just not be this incompetent. It just doesn't compute in my brain."

      *sigh* Quoting somebody earlier in the thread:-

      I've been coding for thirty years and spent fifteen of those as a data recovery engineer and writing DR software. These things are what experience is made of.

      In other enviroments with highly destructive managers however, "You know, these old guys cost us three times more than these new guys we pay the minimum wage to, and they keep arguing that things I tell my staff to do are irresponsibly dangerous and demand that we do things slowly because they say it's "safer". Let's make them all 'redundant' and get a bunch of kids to do the job, they can google anything they don't know these days and their far faster."

      12 Months later. What, you mean that you accidentially deleted all of the servers?!?!! Wasn't somebody supervising... Ah. Well, never mind just do a press release blaming a DDOS/hackers/global warning and and recover the backups. What, you mean we that the backups were in the cloud and deleted with the live enviroment!?! Shit. Well, how do we recover from this? You don't know? Then what do we do?!

  8. batfastad

    123-reg

    I put these bozos in the same bracket as Farthosts. I have migrated many customers from both providers over the years. Not so much for reasons of reliability (although both have been shaky at times) but the customer service was always dreadful and lack of many technical features.

    I definitely understand choosing cheap providers because of cost - if you won't die over a day or so of downtime then thats a perfectly valid decision to make. But if downtime is going to cost your business serious money, it's probably better to not be using budget mass-market web hosts where you are one account of 10,000 and instead have a managed service, or better still run from multiple providers. Compare the loss due to downtime vs the cost of improving availability.

    One thing I would say is separate out your DNS hosting, domain registration and web hosting. Easier to juggle things around in times of brownout. And if the data is important to you, sort your own backups.

    1. Danny 14

      Re: 123-reg

      Host your own stuff. At least you know it is backed up.

    2. cycas

      Re: 123-reg

      Do you have a recommendation for DNS hosting? I don't host with 123reg but I do have domains with them. I've have moved them before this if everything else I try didn't seem to be worse.

  9. Anonymous Coward
    Anonymous Coward

    And that, ladies and gentlemen ..

    .. is what happens when you go for the lowest bidder.

    Exactly that. Ditto for "free", by the way, because that just suggests you're paying with something you have far less control over. If you pay with money you can at least stop payments.

    Having said that, most small business don't even know what sort of risk they are running when they use services like this. The ones that survive this will have learned a valuable (sorry about the pun) lesson.

  10. Anonymous Blowhard

    " it has 3.5 million domain names, and over 1.7 million websites"**

    (** before running the script, now it's 3.1 million domain names, and nearly 1.2 million websites)

    1. Danny 14

      Down to 600k now, the script was running single threaded.

  11. cd / && rm -rf *
    Facepalm

    Too little, too late.

    A new platform will be available by the end of the year for customers which we will provide self-managed and automated snapshot backups

    Horse, meet stable door. Stable door, meet horse.

    1. Anonymous Coward
      Anonymous Coward

      Re: Too little, too late.

      Sorry, the horse has just popped out to show the cow how to levitate.

      1. Mark 85

        Re: Too little, too late.

        No horses needed. Look to the pigs instead.

    2. wolfetone Silver badge

      Re: Too little, too late.

      "Horse, meet stable door. Stable door, meet horse."

      I think the problem here is that the Horse never met the Stable door. If it did, none of this would have happened.

    3. I ain't Spartacus Gold badge
      Devil

      Re: Too little, too late.

      There's nothing wrong with closing the stable door after the horse has bolted. So long as there are going to be more horses along in future. At least you'd hope people might now have learned something...

      So the correct quote should be: "Management, meet stable door. Bang! Ouch! Stable door, meet management. Bang! Ouch! Now, have we learned our lessons? No? OK. And again, management, meet stable door. Bang! Ouch! And for some variety, management, meet anvil! Clang! Next, pair of bricks...?

  12. John Crisp

    I mean seriously 123 customers.... you didn't have backups... AND trusted your provider..... ?????

    Doh.....

    1. Alister

      I mean seriously 123 customers.... you didn't have backups... AND trusted your provider..... ?????

      Doh.....

      But isn't that just what all these vendors keep telling you: no need for on-premises backups, it's all in the Cloud(TM)

      This is why SMB Managers keep falling for it.

      1. Danny 14

        Ive arxhived the story for when the brass decide to try and cut my budget because the cloud is cheaper.

        1. Adam 52 Silver badge

          In this case the customers didn't pay for a managed service. They bought a cheap service that is low resiliance and explicity not backed up, and got what they paid for.

          If your brass want to go cloud then you can, and if you do it right achieve much better availability than almost anything else, do it wrong and you'll be unreliable - just like hosting yourself.

        2. raving angry loony

          The brass won't listen, because they're more likely to hear the salespeople not the technical experts. It's been the case for decades, and will always be the case, because brass almost invariably comes from the sales side of the fence, and they only believe their own brand of lying sacks of shit.

          1. Destroy All Monsters Silver badge
            Trollface

            18/04 never forget!

            So April 18 will henceforth be remembered as "CLOUD DELETION DAY"

    2. grumbley

      Even those who had backups - guess what? The backups got deleted too!

      15 days too late for an April Fool joke!

    3. John Sanders
      Holmes

      Elemental

      Backups you say?

      Nah, the cloud never fails... duh!

      Now, if you'll excuse me, I have to write some more CRON jobs...

      1. Just Enough
        Boffin

        VM <> Cloud

        What's really scary is the number of posts here that equate a VM to "the cloud".

        They are not the same thing, people. If you think they are the same thing then you've missed half the point of "the cloud".

        1. Destroy All Monsters Silver badge

          Re: VM <> Cloud

          Please enlighten us, "Just Enough", what are we missing?

  13. Martin hepworth

    It's in the cloud

    I pay money, they must be backing stuff up... did you actually check?

    1. d3vy

      Re: It's in the cloud

      Actually it's clear that backups are the users responsibility, the problem is that the backup solution provided backs up to the server itself...

      I've got offsite backups, run daily so worst case I've lost 3 hours of data.. The bigger issue for me is that if I reimage and restore service they will overwrite it when they recover my original data.. And that data may be corrupted so it's better for me to wait it out and see what the manage to recover before restoring the backups.

      Though restoring to a different provider is an option... I'm definitely mirroring with someone else.

      1. Danny 14

        Re: It's in the cloud

        What i would do is restore your backup to a competent hosting company. Then get your dns tramsferred and backip/restore as appropriate if they ever get your data back (wishful thinking )

        1. d3vy

          Re: It's in the cloud

          Danny, more or less what's happening. :)

        2. Dr. Mouse

          Re: It's in the cloud

          Personally, I would say that it is the customer's job (for these services, low cost and unmanaged) to ensure there are reliable backups, just as it would be with self-hosted systems.

          HOWEVER, 123-reg's failure here is a clear case of negligence, probably gross negligence, and I would expect a clear cut case in court. If I did this as a contractor, I would expect to be sued, whether the client had backups or not.

          Also, this sort of service is often used by non-technical people. They don't understand the risks of not having a backup, or expect the service provider to back up for them.

          Any professional worth his salt running anything on this kind of server will have a robust backup and DR scheme, and will have tested restoring the backup on both another system from the same SP and on alternative systems. The only people outside 123-reg I can't feel any sympathy for are those IT professionals who lost data through neglecting backups. They should hang their heads in shame. While in this case the problem came down to human error, there are numerous failure modes which are down to dumb luck (multiple disk failures, natural disasters, etc) which they should have been covering off with a decent backup & DR plan.

          1. To Mars in Man Bras!
            Facepalm

            Re: It's in the cloud

            *"...Any professional worth his salt running anything on this kind of server..."*

            I'd question whether there was anything "professional"... or "salt-worthy" about anyone who does this stuff for a living and hosts his clients' sites on the internet equivalent of the Pound Shop.

  14. kain preacher

    Wait this is real? * could of sworn I read a similar but hoax story on el reg.

  15. grumbley

    I Ran A CRON Job On My Bank!

    Has anyone ever heard of Identify - Archive - Delete with the delete portion only being done when you are sure the data is no longer required, happens all the time in the simplest of environments like email programs and the ubiquitous windows Trash Can.

    The result of their script should have been to mark for delete only! Where did these clowns train?

    My business is in ruins today and they cannot even give me a straight answer.

    Am I surprised? I shouldn't be! I was going to leave them last time they lost my dedicated server, but then I thought well at least if I put it all on a VPN then there will be backups and disk images - How wrong I was. I will not be rebuilding anything on 123-reg. I will spend my efforts and money elsewhere. I will also be seeking legal advice.

    And the final cheek was that they tried to collect their Direct Debit for my VPN today. Thankfully I have recalled it, bearing in mind that the payment is monthly in advance I think I have the right, don't you?

    I'll just tell them I ran a CRON job on my bank!

    1. Anonymous Coward
      Anonymous Coward

      Re: I Ran A CRON Job On My Bank!

      Fuck, yeah, I'm been running my bank on dees clowns 123 an' now my customers said that 'is check is bounced an' 'e can't pay nobodies.

      I'm gonna take my bank 'n shit some place better. Why these gangsters charging me £££ for this.

      Bank of AnonCow is changing hosterz!

      1. JoshOvki

        Re: I Ran A CRON Job On My Bank!

        Better off than with HSBC!

    2. Anonymous Coward
      Anonymous Coward

      Re: I Ran A CRON Job On My Bank!

      So you didnt think of creating offsite backups after 123 lost your dedicated server last time?

      Me thinks you need to rethink your setup if its that critical!

  16. Mr_Pitiful

    Strange email from them today

    I got an email advising that a site renewal was due

    Site was probably moved 5 years ago

    The date for site closure in the email was in May 2012

    Have their email servers suffered a timewarp?

    1. 404

      Re: Strange email from them today

      Somebody dusted off tapes from 2012... 'Oh Hey look what I found!' lol

      1. BongoJoe

        Re: Strange email from them today

        Those 2012 tapes will be the back ups...

    2. d3vy

      Re: Strange email from them today

      So.etjing went seriously wrong internally with them today.. I tried to log into support 4-5 times and got a different persons details and tickets every time.

      Called them to advise and they sorted it pretty quick.

      Also had a brief outage because they updated their DNS today.. Apparently.

  17. Leeroy

    Backup

    I'm not sure if they offer an unlimited bandwidth vps package but if they do then you should be running an automated backup at least once a week !

    I had the same thing happen to me and was backing up the entire Vps every other day. Downtime about 12 hours including phoning every customers and backing up their locally cached email that was outside of the backup window. All I lost was some server logs and sales that may have happened in the downtime. Needless to say the backup is now daily.....

    1. d3vy

      Re: Backup

      It is unlimited, I push 5-10GB of backups off various VPS every night to a FTP in the office... I've potentially lost 3 hours worth of data, so not terrible.

  18. TeeCee Gold badge
    WTF?

    Hmm.

    .....this isn't the first time 123 has lost our data in the last few months......

    Fool me once.......(!)

    This is the sort of thing where you really shouldn't give second chances.

  19. Alan Brown Silver badge

    Soo....

    A service explicitly not backed up - and the customers had no backups.

    I bet they'll survive this even if some of the customers don't.

    However.... the discussions here do bring back the point about why tape is great. Having your "backups" be someplace the bad guys can delete them means that the bad guys WILL delete them (This has happened on several occasions).

    With major financial institutions now decreeing the cloud is good enough for backup, whose head will roll when the inevitable happens?

  20. BlueAdmiral

    123-reg are not the first and not the last to have this issue!

    They are not the first and they will not be the last to have something like this happen. Any one who has a business or mission critical services should have either a backup or pay for backup services. It's not rocket science you make sure that really valuable stuff is protected!

  21. Daniel Voyce

    Yeah

    But how sacked is that one dude - seriously?

  22. Anonymous Coward
    Anonymous Coward

    The Cloud...

    Other people's computers you have no control over.

    1. wolfetone Silver badge

      Re: The Cloud...

      Plenty of companies on Twitter offering to host the websites of victims' on their "rock solid cloud" hosting.

      It's all about the money, money, money.

  23. vmistery

    This situation is exactly why I sync all of my data between two different hosts and with two different suppliers. I then have a load of FTP space where the VMs are archived just in case. It is a pain to set up and is not fun to pay for, but it protects me against other peoples negligence or going bust, major outages etc. You really must only trust yourself if something is so critical to your business because although it is their fault it is your problem to deal with your customers and they will be asking you why you didn't keep a backup.

  24. Halfmad

    Am I missing something here?

    I wonder if the staff member who wrote the script still works there or if someone new has come along and tweaked it do to additional stuff and stuffed it in the process.

    I've seen this before when I worked in IT support, normally the new guy checked it does the NEW functions properly but doesn't double check the old ones still work, then runs it on live servers.. then we get another new guy..

  25. Anonymous South African Coward Bronze badge

    A bit late as commentard, but better late than never, eh?

    As sysadmin I hate running timed delete jobs, I prefer to do deletes manually.

    Because boo-boos can happen.

    Rather let the script do backups etc, and so on, that is needed, but when it comes to deleting (or dropping databases), rather do it manually.

    Because Mr Murphy loves to do on-the-fly coding as well.

  26. Seajay#

    You get what you pay for

    Go to the 123-reg site and the first thing you see is

    "Search for your perfect domain from only 99p"

    That should be telling you, this is cheap and cheerful web hosting. Which it is. If your website is business critical and you build it on that without any thought to backups, this is what happens. Even my Wordpress vanity site is backed up to my home computer regularly, where I've checked that I can restore to a local XAMP server. There's nothing of real value there at all, but that sort of system is surely just a habit for anyone who has worked around computers for a while.

    There's nothing wrong with having your website hosted by an outside agent (which you can call the cloud if you must). But if you've got a business that depends on the web then you need to have someone *on your staff* who understands enough to at least know what questions to ask.

    1. Anonymous Coward
      Anonymous Coward

      Re: You get what you pay for

      Go to the 123-reg site and the first thing you see is

      "Search for your perfect domain from only 99p"

      That should be telling you, this is cheap and cheerful web hosting.

      err no, that would be telling you it's cheap and cheerful domain names. You need to look at the hosting packages to determine what sort of hosting company they are...

  27. CAPS LOCK

    Alternative low cost hosters?

    I'm not a 123 customer[1] but I'd like to know who the technorati use...

    [1] Vidahost.

    1. Seajay#

      Re: Alternative low cost hosters?

      Three possible answers:

      1) Hard core nerd - Home server(s) with UPS and redundant network connections

      2) Professional (if my site goes down I can't pay the mortgage) - AWS or similar, spread across availability zones, tested backups, tested failover to a static site served by a CDN.

      3) I'd like my site to stay up but without spending £10k plus - Use any old cheap dodgy hosting company but ensure that you have regular tested backup (stored somewhere other than the dodgy hosting company) which you can upload to somewhere new if it all goes Pete Tong.

      1. wolfetone Silver badge

        Re: Alternative low cost hosters?

        Personally I use Digital Ocean and look after the VPS's myself. They are then backed up to an Azure storage cloud. I have a VPS with OVH that is dormant but is configured with my clients accounts, so that if DO do a Linode and go AWOL, I can restore the sites from Azure and switch domain names for prolonged downtime.

        I've never had to do this though. Thankfully!

  28. Anonymous Coward
    Anonymous Coward

    Sickapedia

    Sickapedia is still down from a web hosting issue :(

    1. d3vy

      Re: Sickapedia

      Yeah that's been down for ages... They said recently their disks were with a data recovery company... Probably did the same thing....

  29. zosim

    "One customer claimed this practice "is frowned upon in the hosting world... I discussed the issue with a few acquaintances who have experience running servers in a commercial environment and all of them were shocked to hear that 123-reg 'automate clean-up of servers'"

    Having worked in mass hosting, the only option for cleanup is automation for large scale platforms and I doubt there's a single large hoster on the planet that doesn't do it, particularly for low-cost hosting. At most large dedicated server companies, provisioning, building and deletion are also automated. There's a real risk, obviously, of mistakes happening but the real surprise is it doesn't happen more often.

    1. Steven Raith

      What I want to know is how they managed to delete live instances.

      I mean, that's basic, basic stuff.

      Is this VM flagged for deletion currently in a running state? Yes. ABORT ABORT ABORT.

      You can automate suspensions (after all, you just need to pause, or down the VM - pick whichever suits your flagging method to determine how agressive you want to be) but you should never, ever authorise the scripted deletion of a VM without checking if it's running first.

      Hell, I'd not even want to automate the deletion of suspended servers personally (Although I'll accept that beyond a certain scale, it's unavoidable), just in case you get the syntax wrong and do a 123-reg, as I suspect that's what's happened here...

      Steven R

  30. Picky
    Unhappy

    Real customer hit

    http://www.bbc.co.uk/news/uk-scotland-highlands-islands-36080667

  31. Anonymous Coward
    Anonymous Coward

    Anon in case the boss reads and recognises ...

    At work (IT services company) we use cloud services, and the boss is actively trying to push everything off-site so he can shut down the server room.

    So far, every discussion regarding backups etc has come down to "SEP" - it's someone else's problem. OK, we're not using budget cowboys like 123, but even the big boys have all had problems. And all it takes is for "an administrative error" to delete your account and it's gone.

    1. Anonymous Coward
      Anonymous Coward

      MS

      Even last week MS had to issue idiot proof guidance as to what IaaS through to SaaS means in terms of what is still your issue.

      And believe me I've had to point this out to senior architects in a big 4 firm, as they hadn't even considered backup till the last moment.

      Anon for obvious reasons

  32. Andrew Barr
    Trollface

    Does it matter

    Surely it doesn't matter what they do now as all the customers will be jumping ship.

    No need to audit and check the scripts, in a few months they should be able to just format the disks and put the hardware on ebay!

  33. Duffaboy
    Mushroom

    I always like it when they state "A small proportion of our customer base has been affected"

    when they really mean, everyone

  34. NotMyRealName
    Unhappy

    It does matter!

    To techies, it may be 'obvious' to create one's own backups, held off-site, etc. etc. And equally 'obvious' to not use the cheapest host on the block (although paying doesn't guarantee 100% infallibility). That kind of thinking is what techies get paid for. But, instead of sneering, can you not spare a thought for the hapless 'loosers'?

    Imagine you've just set up your own small business. Money is tight; you'll have to survive on your savings until (if?) the business takes off. You may know a lot about building the proverbial better mouse-trap. That doesn't mean you know owt about IT, other than the fact that any business nowadays needs a Web presence. And, along with everything else in the start-up stage, that has to be done on a shoe-string.

    Let's assume you're one of the lucky ones: your business does take off. Now, your most pressing problem is dealing with, and servicing, the influx of orders. (Happy customers = repeat customers, and all that.) You're working very long hours to keep the business afloat. You may even be making enough profit to start paying yourself! One thing you aren't fretting about is your Web presence because, thus far, it's been working fine. And, as everyone knows, if it ain't broke you don't fix it ...

    C'mon, guys. Where's your imagination, your empathy?

    1. cycas

      Re: It does matter!

      Sadly, sympathy butters no parsnips, the most important thing someone affected by this can really take from it is that in future they SHOULD have backups and that this is not something you can economise on or forget about or put off till later. :-(

      I haven't read any comments that I would say are sneering, but what can you say to someone who has just lost all their data and didn't back it up, other than 'backups are good' which they already know?

  35. MilesWeb

    This time consider backup says MilesWeb.com

    Everyone using web hosting service should strictly consider remote backup plans. In case of such disaster atleast your important data would be safe on the backup server. Many hosting providers strictly advise the customers to opt for a separate backup plan. Many affected customers of 123-reg would prefer to switch to a new provider and hope this time they opt for backup solution.

  36. Anonymous Coward
    Anonymous Coward

    123-reg web hosting - shocking customer support!

    123-reg has become a joke. As a customer for 6 years I have recently had my web site hacked and scripts added in the early hours of the morning on 3 occasions in two months. I have changed 123-reg admin/FTP and Wordpress passwords on three occasions. Now my website has been suspended since 3rd of May and is still down on the 23rd of May although I have followed all 123 suggestions and have deleted all content and restored from a safe backup. I have replied by ticket and rang three times. I was told over a week ago this was being resolved with 24 hours. Since then I have phoned an addition 3 times, each time I waited 20 minutes with no answer. Have they packed up? I have worked in IT for 20 years and never witnessed such a poor support/helpdesk system. When I first managed to speak to their support team I pointed out my ticket was raised by them on the May 3rd and I had replied multiple times and they still hadn't got back to me and it was now the 13th of May. I was told that every time I submitted a new response to the original ticket I was put to the back of the queue, have you ever heard such madness??? So they do nothing and the obvious question you ask is what's the progress of this ticket? and they put you to the back of the queue. Also seems strange how it keeps getting hacked so soon after I've changed all passwords and nobody else has access them, I even store them in a secure area on my system. I only use one PC to make changes and it's never had a virus as I'm OCD with virus protection. Seems like an inside leak...Time to abandon the Titanic!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like