back to article Catastrophic 123-reg VPS cockup deletes Ross County FC website

Ross County Football Club's website was among those deleted amid a web-hosting company 123-reg's mega cock-up. Virtual servers vanished for a large number of customers after the hosting firm ran a script containing a catastrophic error at 7am on Saturday 16 April as part of its clean-up process on the 123-reg VPS platform. “ …

  1. djstardust Silver badge

    Oh dear

    I can't believe a business of that magnitude would use a dodgy tuppence hapenny Web host like that.

    Thousands of ticket sales down the crapper to save a couple of quid.

    Bet they're looking for a new It guy......

    1. Anonymous Coward
      Anonymous Coward

      Re: Oh dear

      I can't believe a business of that magnitude would use a dodgy tuppence hapenny Web host like that.

      We're talking about a team with an average attendance of a tadge over 4,000 here, not Manchester United. :) A decent VPS with a couple of other hosts on the server is probably perfectly adequate for their needs.

      1. Lee D Silver badge

        Re: Oh dear

        4000 x the price of a ticket x every fixture over a year.

        That's a lot of money to be losing on the risk of having a cheap junky website.

        1. Seajay#

          Re: Oh dear

          "4000 x the price of a ticket x every fixture over a year.

          That's a lot of money to be losing on the risk of having a cheap junky website."

          That would be a lot of money to be losing but I think the relevant calculation is

          Number of tickets likely to be sold in a 4 day period

          x Proportion of fans who refuse to use the phone, buy them in person or come back to the site 4 days later

          x price of a ticket

          Probably more than half of the attendance are season ticket holders. They probably have a home match every other week so the maximum realistic number for 4 days sales would be 2000 * 4/14 = 570

          maybe 9/10 people who will try the phone or try again later. You're looking at 50 lost ticket sales at about £20 each.

          Best guess £1000

          Bearing in mind that this was the worst case (they were one of the unlucky customers whose websites were taken out and they have a game against Celtic soon) I'd say they were entirely sensible to pick a cheap hosting option.

      2. Anonymous Coward
        Anonymous Coward

        Re: Oh dear

        1.7M sites on 115K servers is not a couple on each server. This is typical, if you take money via your site, sales which you rely on then a tenner a month doesn't cut it. <- that's a full stop.

    2. GavinC

      Re: Oh dear

      Ross County FC? I'm not sure what kind of magnitude you refer to, but I'd certainly imagine them to be using a cheapish host.

      But as someone that's had VPS's in the past, it's always been explicitly mentioned in the T&Cs that you are responsible for your own backups. Any one failing to do so themselves is just asking for trouble.

    3. Vince

      Re: Oh dear

      I can't believe (well I can entirely believe) they didn't read the terms and conditions.

      Clause 1.5 (not even buried anywhere):

      "1.5 The "Back-ups" Add-on provided by 123-reg includes access and maintenance however Customers are solely responsible for ensuring that full backups have been made."

      The good news is that even if 123-reg pay you, you'll get:

      "9.1 THE TOTAL SOLE AND EXCLUSIVE REMEDY AVAILABLE TO YOU AS THE RESULT OF ANY BREACH OF THIS AGREEMENT, NEGLIGENCE, OR ANY ACTION OR FAILURE TO ACT WHETHER INTENTIONAL OR OTHERWISE SHALL BE LIMITED TO THE TOTAL AMOUNT OF FEES PAID BY YOU TO 123-REG IN THE TWELVE MONTHS IMMEDIATELY PRECEDING ANY ALLEGATION OF THE AFOREMENTIONED ACTIONS. "

      ...so if you had one of those bargain £9.99 servers, you can expect £120.00 or so tops.

      1. TeeCee Gold badge
        Facepalm

        Re: Oh dear

        Unfortunately for 123-Reg, neither of those clauses is worth the paper it's printed on.

        Damages and penalties for breach of contract are what a court decides they are, not what your weasel clause says. That's only in there in the hope that the customers won't bother sueing when they see them....

    4. myarse
      Happy

      Re: Oh dear

      I'd expect the whole IT department to get the chop.

      1. a_yank_lurker Silver badge

        Re: Oh dear

        Do they even have an IT department? From the comments it sounds like they are shoestring, penny pinching operation out of necessity.

        1. Triggerfish

          Re: Oh dear

          Do they even have an IT department? From the comments it sounds like they are shoestring, penny pinching operation out of necessity.

          The football club or 123 reg?

      2. Anonymous Coward
        Anonymous Coward

        Re: Oh dear

        Yeah, the whole department He must be bricking it.

        Do they even have the internet in Dingwall? I am surprised they wear shoes quite frankly.

    5. d3vy Silver badge

      Re: Oh dear

      123 are one of the biggest hosts in the UK.. They are not the cheapest.. Quite middle of the road actually.

      Regardless of how good or bad they are you don't expect your hosting company to effectively trash your server by accident.

  2. John Sanders
    Holmes

    >> The company has since revealed that fault affected 67 servers out of 115,000.

    Hmm it seems included in these 67 was the server farm that does the cheap VPS.

    The remaining 114933 servers were either other systems or "dedicated" I guess.

    1. Anonymous Coward
      Anonymous Coward

      Odd looking numbers

      The ratio of physical to VPS hosts doesn't seem credible to me.

      Assuming they are deliberately conflating 67 physical servers and the number of VPS servers because it 'looks better'.

      Equally if they actually do have 115k physical servers (of which 67 were affected) they need to learn about contention ratios because that seems suspiciously close to one physical for each VPS host.

      I tend to suspect the former.

    2. Alister Silver badge

      I think you are misunderstanding.

      I thought is was 67 VMs that were lost, not physical servers.

      1. Lee D Silver badge

        67 appears just right for:

        - Okay, the script is good?

        - Yes, it's good.

        - Tested it.

        - Er... yeah.

        - Okay, press it.

        - See, it's fine.

        - Okay. I'll leave you to - erm. What's that?

        - What's what?

        - That machine name that just flicked past, scroll back a bit.

        - Where?

        - CTRL-C, CTRL-C, STOP IT!

        - What?!

        - <pushes operator aside, grabs keyboard> You idiot!

        - What?!

        - Those were live!

        - Er... Oh.

        - You go and tell the helpdesk to expect calls. I'll see what the damage is.

    3. Wibble
      Mushroom

      I call that 123 bollocks.

      There's no way on earth that they have 115 THOUSAND SERVERS. At 42 servers per rack, that's 2740 full height racks - if you can get 50 racks in a double-decker bus, then you'd need 55 London buses, consuming the floor space of three micro-Wales. They are talking about VPS'

      The 67 SERVERS would be the things they ran this untested script on. With each one hosting, [finger in the air] 20 VPS', this would be 1200 VPS's deleted.

      Which is why I personally know of three different VPS' -- including my WIP dev VPS -- which has been deleted.

      1. Ian Watkinson

        No I'd say that's about right.

        115,000 WEB servers running on 67 virtual servers.

        1716 per server.

        Seems about right, it's not like most of the websites are going to generate much traffic or load, and if 123-reg are clever with their capacity, they'd move the livelier sites around to balance them on the 67 servers.

        FWIW.

        123-reg bad, no back up prior to running a destructive script and no roll-back or fall-back plan.

        Customers bad for not having their own backups. It takes all of about 1 day to set up an sftp pull to get their site down stored locally. To then put back up couple of hours later should 123reg you know be 123reg again...

  3. Daniel Hall
    Alert

    And where was...

    Their local backup of the website?

    Surely it would have been quicker knowing the issue is quite large, to re-point the domain and setup hosting elsewhere?

    Propagation only takes 10mins to 24 hours from experience working at Fasthosts.

    1. Bob the Skutter

      Re: And where was...

      I'm confused as to how so many people lost websites.

      All mine are edited locally and then uploaded to live server.

      Who maintains only a live version of a website?

      1. 2+2=5 Silver badge

        Re: And where was...

        > I'm confused as to how so many people lost websites. All mine are edited locally and then uploaded to live server. Who maintains only a live version of a website?

        To be fair, we've only heard of a few websites where they had no backup. There are probably many others (the majority I would hope) who do have their own backups but, because they are getting such piss poor information from 123-reg, don't know whether to restore themselves or wait longer.

        I'm not a customer, but from what I've read, I've no idea whether replacement VPSs are up and running, with original domain names, usernames and passwords so that customers can just log in and restore data only? A previous El Reg article did mention something about 123-reg suggesting people set themselves up again but no mention of whether that would incur double-billing or how 123-reg intended to deal with it.

      2. BebopWeBop Silver badge

        Re: And where was...

        I'm confused as to how so many people lost websites. All mine are edited locally and then uploaded to live server. Who maintains only a live version of a website?

        Well that is fine for a static site. However if the site is in some way changing - based on, say comments from fans, ticket transactions etc... then no matter ho many times you store the static content, your other information is at the mercy of 123

      3. maffski

        Re: And where was...

        I'm confused as to how so many people lost websites.

        The system became that overloaded that re-initialising my VPS instance took about twelve hours, until that's complete there's nowhere for you to restore your backup into.

        1. d3vy Silver badge

          Re: And where was...

          I got an email saying mine was ready but I still can't get to it...

    2. P. Lee Silver badge

      Re: And where was... Their local backup of the website?

      I think the first article mentioned that the "backups" were online rather than offline so the rm -rf {foo}/{bar} took them all out at the same time. No news on whether the script ran as root or an unprivileged user but the backups were held under the same user privileges. That's Agile DevOps!

      This is where automation is a bonus if you use something well tested. The point about automation is that you get people who know what they are doing to encapsulate their expertise and experience (check your parameters before you delete) which can then be safely reused by those less skilled.

      You deployed a script straight into Production? Smack! The rm -rf {blank}/{blank} legend is pretty much the first thing you read if you read any unix shell scripting book. I wonder if 123-reg is reconsidering whether hiring really cheap admins is a good business decision?

  4. Anonymous Coward
    Anonymous Coward

    And still people will prefer to outsource their core business to others...

    ...penny wise, pound foolish.

    Outsourcing have its time and place, but when it comes to core IT, don't. Rather pay for dedicated hardware, communications and experienced personnel than trying to earn big bonuses just because you've cut IT expenditure very thin.

    1. Adam McCormack
      Coat

      Ross County's core business is not web hosting...or technology

      1. wolfetone Silver badge
        Trollface

        Or football.

      2. Anonymous Coward
        Anonymous Coward

        Core business

        Yes. But the real question is not "what is the core business of Ross County" but "can Ross county continue it's core business in the absence of it's website" ?

        Put that way then given the subsequent issues (i.e. lack of revenue !!!!) it seems fair to suggest ROss County should have put more stock in their web provision.

        If we compare that situation to a metal bashing outfit from the 70s, there are parallels. Said metal bashers could probably carry on if their telephone was out for a bit. Or if there was a postal strike. They could probably limp on for a day or two without potable water and/or sewage facilities. However, if they lost their electricity or gas supply .... which is why portable generators and oil fuel are used as standbys . Not universally, but the principle is there.

        Which leads me to a startup idea of a backup hosting company who are ready to spin up as a hot standby if your primary provider goes all 1-2-3.

    2. Anonymous Coward
      Anonymous Coward

      Their core business is selling tea (1.60) and haggis pie (2.20) to the supporters.

    3. Anonymous Coward
      FAIL

      So your saying they should have a dedicated web developer, a sys admin, a network engineer and a security expert (minimum)?

      Add to that a server, UPS, generator, firewall,switch, a rack to put all this in, a comms room to put the rack in, oh lets not forget offsite back up + plus full DR redundant equipment.

      Well, that's their entire budget gone for the year.

      1. theblackhand

        Re: So your saying they should have a dedicated web developer, a sys admin, a network engineer and a security expert (minimum)?

        Depends - as long as they can kick a ball they can be part of the team...

  5. This post has been deleted by its author

  6. Anonymous Coward
    Anonymous Coward

    Once again, a clear demonstration that a belief

    that "IT" is not a core activity is proven wrong.

    One of the most powerful things I have ever read in El Reg was a comment during the RBS outage (which one ?) where it was noted that banks nowadays were really IT departments who happened to move money as a sideline.

    When you look at it that way (i.e. how integral IT is to *every* activity an organisation undertakes) then the wisdom/value of offloading/outsourcing it suddenly changes dramatically.

    I have finally managed to get our board to stop thinking of "IT" as a cost, after they were forced to admit that without it, it would be impossible (not "difficult", not "take longer") to conduct business manually.

    1. Anonymous Coward
      Anonymous Coward

      Re: Once again, a clear demonstration that a belief

      Reminds me of a (terrible, terrible) company I contracted for a few years back. They ran a 24-hour call centre operation and begrudged every penny they spent on IT seeing it purely as a cost.

      But, as you pointed out, without the IT they had NO business at all.

      I was glad to leave when my contract expired.

      1. Doctor Syntax Silver badge

        Re: Once again, a clear demonstration that a belief

        "without the IT they had NO business at all."

        I've often thought that the best way to deal with PHBs who want IT to justify itself is to offer to switch it all off for a day to see what happens.

        1. Anonymous Coward
          Anonymous Coward

          Re: Once again, a clear demonstration that a belief

          "without the IT they had NO business at all."

          Well if you exclude telephone sales or turning up at the gate.....you know like they did before online ticket sale.

    2. Anonymous Coward
      Anonymous Coward

      Re: Once again, a clear demonstration that a belief

      Come on, a bank is a bit of a different proposition than a small football club. Outsourcing their website's server makes a great deal of sense - a reliable host will probably do a damn sight better job of maintaining it than the kind of internal IT bod they could afford. For them, IT is a cost, just like the electricity for their floodlights and pie van, the water for their loos, and other vital utilities!

    3. disgustedoftunbridgewells Silver badge

      Re: Once again, a clear demonstration that a belief

      For a club the size of Ross County FC, there a better than even chance that the whole IT department also happens to be the left-back for the first team.

  7. Tezfair
    Thumb Down

    cloud goes pooooof

    Shows how easy 'your' files in the cloud can be removed overnight.

    That aside, (I have no clients nor interest in 123) i'm surprised 123 are saying 'tough' to those without a backup. As it was their fault they deleted the files, they have a moral obligation to undo that damage.

    1. John G Imrie Silver badge

      they have a moral obligation to undo that damage.

      A moral obligation is worth less than the paper it's written on. Ask a Politician.

    2. Martin hepworth

      Re: cloud goes pooooof

      I think theres a level of transparency from the folks like 123reg that needs to be increased so the smaller folks selling stuff via sites and the like are completely aware of risks and issues they could face.

    3. David Neil

      Re: cloud goes pooooof

      If they were never contracted to provide backups, and clearly mentioned that in the terms, then surely it behooves the customer to understand what they have actually bought for £9.99

      Morally does not mean I get to cut costs to the bone and I can guilt trip someone into giving me service I knowingly decided I didn't want to pay for.

      1. d3vy Silver badge

        Re: cloud goes pooooof

        "Morally does not mean I get to cut costs to the bone and I can guilt trip someone into giving me service I knowingly decided I didn't want to pay for."

        You know what else I didn't pay for? Them to delete my VM... They did that all onntheir own.

        I have backups but no where to restore to as they can't provision a new machine.

        So now I have to move to a new provider and try to to recover the annual payment that's already been taken.

  8. tekHedd

    rm -rf ...

    I can't help wondering if the number of affected customers on those 67 servers is just about 1500...

    "It was a fake, a troll. Really!"

  9. Anonymous Coward
    Anonymous Coward

    tbh why don't these businesses have their own backups too?

    1. Anonymous Coward
      Anonymous Coward

      To which the answer would be what's a backup, followed closely it's in the cloud anyway.

    2. d3vy Silver badge

      A lot do... But have no where to restore it.

      Or lost the backups because they were also hosted with 123....

  10. Anonymous Coward
    Anonymous Coward

    Victim blaming

    The idea that a two-bit football club should do its own hosting is like saying you should learn to pilot an airliner because your flight was delayed. The problem here is the hosting company screwed up in a way few would have believed before it happened. The customer should have an expectation that the company has the expertise to operate properly. Perhaps a national inspectorate and rating system

    1. wolfetone Silver badge

      Re: Victim blaming

      "The customer should have an expectation that the company has the expertise to operate properly."

      Yes, because you should expect to get the same service from 123-Reg at £14.99 a month as you would do from Rackspace at nearly £700 a month.

    2. P. Lee Silver badge

      Re: Victim blaming

      The problem is that if you use a service/asset, you should understand what you are using and at very least the T's & C's.

      If the football club had a website, someone should have assessed what the cost of the website and/or data going away was and made contingency plans. If something is important to you, make your own plans to preserve it. Worst case, have some USB sticks with a copy of the website on it, or pay a different provider 14.99/month to host a manually replicated site, in case the whole provider goes away.

  11. Doctor Syntax Silver badge

    I don't know the ins & outs of this particular site* but I suspect in a lot of cases the business has no IT knowledge whatsoever. They found somebody local who could "do" them a website for some amount of money. The somebody has then arranged to host the site and moved onto the next client. The client may not even have seen the T&Cs, much less read them, much less understood them. It wouldn't surprise me if there were e-commerce sites there where the business's only copy if its entire transactional history is the website's database complete with customers' credit card details.

    *It's football - my personal view is that anything involving 22 men chasing a bag of wind up and down a field could be removed from the planet with no loss whatsoever.

  12. To Mars in Man Bras!
    FAIL

    I Was Shocked...

    ...to see how many Design Agencies are amongst the disgruntled customers laying siege to 123-Reg's Twitter account. You'd really expect industry professionals to know better. As I wittily put it myself, "It's a bit like finding out your mechanic buys his tools at the Pound Shop."

    I feel really sorry for the small business clients who outsourced the maintenance of their web presence to such agencies though. I'm sure, in a lot of cases, the client was being charged a lot more than a tenner a month for "hosting" —little knowing that their design agency was renting server space from a bargain-bucket outfit like 123-Reg and pocketing the difference.

    1. Triggerfish

      Re: I Was Shocked...

      I dunno, last web design agency that was in our office mainly seemed experts on marketing speak, and that sort of general bullshit. I mean don't get me wrong the sites nice, but coding wise HTML and CSS with a bit of jscript is not exactly hardcore techy knowledge, why would you expect people who mainly trained in graphic design and marketing to know the other aspects of important IT business, like backups, making sure the hosts are solid, etc. The lot who came in to sell the website did not strike me as techies, they were salespeople and designers.

      Not saying there are not good companies out there or websites that are not complex, but your average business owner tends to fall for the sales people and buzwords far more, so thats what you get when they sell you their product, and most cheap hosting outfits provide some sort of shopping cart type sales device that can be implemented.

      It's very hard to convince the beancounter there is a difference between paying say a couple of hundred quid and a couple of grand more for a site that looks to their eyes the same.

  13. Anonymous Coward
    1. Ally Bee

      Says the Anonymous Coward .....

  14. d3vy Silver badge

    67 out of 115k... That 115k is their entire server estate isn't it... Where as the 67 affected are hosts for VMs...

    So how many VMs did they delete - it was more than 67.

  15. d3vy Silver badge

    Also, I've not had any contact with 123 other than that which I have initiated... So they are hardly keeping customers in the loop.

    And the advice they gave to rebuild the machines fell on its arse when I did it, got the email to say the new server was up and ready... But can't connect... And support are not answering.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019