Oh dear
I can't believe a business of that magnitude would use a dodgy tuppence hapenny Web host like that.
Thousands of ticket sales down the crapper to save a couple of quid.
Bet they're looking for a new It guy......
Ross County Football Club's website was among those deleted amid a web-hosting company 123-reg's mega cock-up. Virtual servers vanished for a large number of customers after the hosting firm ran a script containing a catastrophic error at 7am on Saturday 16 April as part of its clean-up process on the 123-reg VPS platform. “ …
I can't believe a business of that magnitude would use a dodgy tuppence hapenny Web host like that.
We're talking about a team with an average attendance of a tadge over 4,000 here, not Manchester United. :) A decent VPS with a couple of other hosts on the server is probably perfectly adequate for their needs.
"4000 x the price of a ticket x every fixture over a year.
That's a lot of money to be losing on the risk of having a cheap junky website."
That would be a lot of money to be losing but I think the relevant calculation is
Number of tickets likely to be sold in a 4 day period
x Proportion of fans who refuse to use the phone, buy them in person or come back to the site 4 days later
x price of a ticket
Probably more than half of the attendance are season ticket holders. They probably have a home match every other week so the maximum realistic number for 4 days sales would be 2000 * 4/14 = 570
maybe 9/10 people who will try the phone or try again later. You're looking at 50 lost ticket sales at about £20 each.
Best guess £1000
Bearing in mind that this was the worst case (they were one of the unlucky customers whose websites were taken out and they have a game against Celtic soon) I'd say they were entirely sensible to pick a cheap hosting option.
Ross County FC? I'm not sure what kind of magnitude you refer to, but I'd certainly imagine them to be using a cheapish host.
But as someone that's had VPS's in the past, it's always been explicitly mentioned in the T&Cs that you are responsible for your own backups. Any one failing to do so themselves is just asking for trouble.
I can't believe (well I can entirely believe) they didn't read the terms and conditions.
Clause 1.5 (not even buried anywhere):
"1.5 The "Back-ups" Add-on provided by 123-reg includes access and maintenance however Customers are solely responsible for ensuring that full backups have been made."
The good news is that even if 123-reg pay you, you'll get:
"9.1 THE TOTAL SOLE AND EXCLUSIVE REMEDY AVAILABLE TO YOU AS THE RESULT OF ANY BREACH OF THIS AGREEMENT, NEGLIGENCE, OR ANY ACTION OR FAILURE TO ACT WHETHER INTENTIONAL OR OTHERWISE SHALL BE LIMITED TO THE TOTAL AMOUNT OF FEES PAID BY YOU TO 123-REG IN THE TWELVE MONTHS IMMEDIATELY PRECEDING ANY ALLEGATION OF THE AFOREMENTIONED ACTIONS. "
...so if you had one of those bargain £9.99 servers, you can expect £120.00 or so tops.
Unfortunately for 123-Reg, neither of those clauses is worth the paper it's printed on.
Damages and penalties for breach of contract are what a court decides they are, not what your weasel clause says. That's only in there in the hope that the customers won't bother sueing when they see them....
The ratio of physical to VPS hosts doesn't seem credible to me.
Assuming they are deliberately conflating 67 physical servers and the number of VPS servers because it 'looks better'.
Equally if they actually do have 115k physical servers (of which 67 were affected) they need to learn about contention ratios because that seems suspiciously close to one physical for each VPS host.
I tend to suspect the former.
67 appears just right for:
- Okay, the script is good?
- Yes, it's good.
- Tested it.
- Er... yeah.
- Okay, press it.
- See, it's fine.
- Okay. I'll leave you to - erm. What's that?
- What's what?
- That machine name that just flicked past, scroll back a bit.
- Where?
- CTRL-C, CTRL-C, STOP IT!
- What?!
- <pushes operator aside, grabs keyboard> You idiot!
- What?!
- Those were live!
- Er... Oh.
- You go and tell the helpdesk to expect calls. I'll see what the damage is.
I call that 123 bollocks.
There's no way on earth that they have 115 THOUSAND SERVERS. At 42 servers per rack, that's 2740 full height racks - if you can get 50 racks in a double-decker bus, then you'd need 55 London buses, consuming the floor space of three micro-Wales. They are talking about VPS'
The 67 SERVERS would be the things they ran this untested script on. With each one hosting, [finger in the air] 20 VPS', this would be 1200 VPS's deleted.
Which is why I personally know of three different VPS' -- including my WIP dev VPS -- which has been deleted.
No I'd say that's about right.
115,000 WEB servers running on 67 virtual servers.
1716 per server.
Seems about right, it's not like most of the websites are going to generate much traffic or load, and if 123-reg are clever with their capacity, they'd move the livelier sites around to balance them on the 67 servers.
FWIW.
123-reg bad, no back up prior to running a destructive script and no roll-back or fall-back plan.
Customers bad for not having their own backups. It takes all of about 1 day to set up an sftp pull to get their site down stored locally. To then put back up couple of hours later should 123reg you know be 123reg again...
> I'm confused as to how so many people lost websites. All mine are edited locally and then uploaded to live server. Who maintains only a live version of a website?
To be fair, we've only heard of a few websites where they had no backup. There are probably many others (the majority I would hope) who do have their own backups but, because they are getting such piss poor information from 123-reg, don't know whether to restore themselves or wait longer.
I'm not a customer, but from what I've read, I've no idea whether replacement VPSs are up and running, with original domain names, usernames and passwords so that customers can just log in and restore data only? A previous El Reg article did mention something about 123-reg suggesting people set themselves up again but no mention of whether that would incur double-billing or how 123-reg intended to deal with it.
I'm confused as to how so many people lost websites. All mine are edited locally and then uploaded to live server. Who maintains only a live version of a website?
Well that is fine for a static site. However if the site is in some way changing - based on, say comments from fans, ticket transactions etc... then no matter ho many times you store the static content, your other information is at the mercy of 123
I think the first article mentioned that the "backups" were online rather than offline so the rm -rf {foo}/{bar} took them all out at the same time. No news on whether the script ran as root or an unprivileged user but the backups were held under the same user privileges. That's Agile DevOps!
This is where automation is a bonus if you use something well tested. The point about automation is that you get people who know what they are doing to encapsulate their expertise and experience (check your parameters before you delete) which can then be safely reused by those less skilled.
You deployed a script straight into Production? Smack! The rm -rf {blank}/{blank} legend is pretty much the first thing you read if you read any unix shell scripting book. I wonder if 123-reg is reconsidering whether hiring really cheap admins is a good business decision?
And still people will prefer to outsource their core business to others...
...penny wise, pound foolish.
Outsourcing have its time and place, but when it comes to core IT, don't. Rather pay for dedicated hardware, communications and experienced personnel than trying to earn big bonuses just because you've cut IT expenditure very thin.
Yes. But the real question is not "what is the core business of Ross County" but "can Ross county continue it's core business in the absence of it's website" ?
Put that way then given the subsequent issues (i.e. lack of revenue !!!!) it seems fair to suggest ROss County should have put more stock in their web provision.
If we compare that situation to a metal bashing outfit from the 70s, there are parallels. Said metal bashers could probably carry on if their telephone was out for a bit. Or if there was a postal strike. They could probably limp on for a day or two without potable water and/or sewage facilities. However, if they lost their electricity or gas supply .... which is why portable generators and oil fuel are used as standbys . Not universally, but the principle is there.
Which leads me to a startup idea of a backup hosting company who are ready to spin up as a hot standby if your primary provider goes all 1-2-3.
So your saying they should have a dedicated web developer, a sys admin, a network engineer and a security expert (minimum)?
Add to that a server, UPS, generator, firewall,switch, a rack to put all this in, a comms room to put the rack in, oh lets not forget offsite back up + plus full DR redundant equipment.
Well, that's their entire budget gone for the year.
This post has been deleted by its author
that "IT" is not a core activity is proven wrong.
One of the most powerful things I have ever read in El Reg was a comment during the RBS outage (which one ?) where it was noted that banks nowadays were really IT departments who happened to move money as a sideline.
When you look at it that way (i.e. how integral IT is to *every* activity an organisation undertakes) then the wisdom/value of offloading/outsourcing it suddenly changes dramatically.
I have finally managed to get our board to stop thinking of "IT" as a cost, after they were forced to admit that without it, it would be impossible (not "difficult", not "take longer") to conduct business manually.
Reminds me of a (terrible, terrible) company I contracted for a few years back. They ran a 24-hour call centre operation and begrudged every penny they spent on IT seeing it purely as a cost.
But, as you pointed out, without the IT they had NO business at all.
I was glad to leave when my contract expired.
Come on, a bank is a bit of a different proposition than a small football club. Outsourcing their website's server makes a great deal of sense - a reliable host will probably do a damn sight better job of maintaining it than the kind of internal IT bod they could afford. For them, IT is a cost, just like the electricity for their floodlights and pie van, the water for their loos, and other vital utilities!
Shows how easy 'your' files in the cloud can be removed overnight.
That aside, (I have no clients nor interest in 123) i'm surprised 123 are saying 'tough' to those without a backup. As it was their fault they deleted the files, they have a moral obligation to undo that damage.
If they were never contracted to provide backups, and clearly mentioned that in the terms, then surely it behooves the customer to understand what they have actually bought for £9.99
Morally does not mean I get to cut costs to the bone and I can guilt trip someone into giving me service I knowingly decided I didn't want to pay for.
"Morally does not mean I get to cut costs to the bone and I can guilt trip someone into giving me service I knowingly decided I didn't want to pay for."
You know what else I didn't pay for? Them to delete my VM... They did that all onntheir own.
I have backups but no where to restore to as they can't provision a new machine.
So now I have to move to a new provider and try to to recover the annual payment that's already been taken.
The idea that a two-bit football club should do its own hosting is like saying you should learn to pilot an airliner because your flight was delayed. The problem here is the hosting company screwed up in a way few would have believed before it happened. The customer should have an expectation that the company has the expertise to operate properly. Perhaps a national inspectorate and rating system
The problem is that if you use a service/asset, you should understand what you are using and at very least the T's & C's.
If the football club had a website, someone should have assessed what the cost of the website and/or data going away was and made contingency plans. If something is important to you, make your own plans to preserve it. Worst case, have some USB sticks with a copy of the website on it, or pay a different provider 14.99/month to host a manually replicated site, in case the whole provider goes away.
I don't know the ins & outs of this particular site* but I suspect in a lot of cases the business has no IT knowledge whatsoever. They found somebody local who could "do" them a website for some amount of money. The somebody has then arranged to host the site and moved onto the next client. The client may not even have seen the T&Cs, much less read them, much less understood them. It wouldn't surprise me if there were e-commerce sites there where the business's only copy if its entire transactional history is the website's database complete with customers' credit card details.
*It's football - my personal view is that anything involving 22 men chasing a bag of wind up and down a field could be removed from the planet with no loss whatsoever.
...to see how many Design Agencies are amongst the disgruntled customers laying siege to 123-Reg's Twitter account. You'd really expect industry professionals to know better. As I wittily put it myself, "It's a bit like finding out your mechanic buys his tools at the Pound Shop."
I feel really sorry for the small business clients who outsourced the maintenance of their web presence to such agencies though. I'm sure, in a lot of cases, the client was being charged a lot more than a tenner a month for "hosting" —little knowing that their design agency was renting server space from a bargain-bucket outfit like 123-Reg and pocketing the difference.
I dunno, last web design agency that was in our office mainly seemed experts on marketing speak, and that sort of general bullshit. I mean don't get me wrong the sites nice, but coding wise HTML and CSS with a bit of jscript is not exactly hardcore techy knowledge, why would you expect people who mainly trained in graphic design and marketing to know the other aspects of important IT business, like backups, making sure the hosts are solid, etc. The lot who came in to sell the website did not strike me as techies, they were salespeople and designers.
Not saying there are not good companies out there or websites that are not complex, but your average business owner tends to fall for the sales people and buzwords far more, so thats what you get when they sell you their product, and most cheap hosting outfits provide some sort of shopping cart type sales device that can be implemented.
It's very hard to convince the beancounter there is a difference between paying say a couple of hundred quid and a couple of grand more for a site that looks to their eyes the same.
Also, I've not had any contact with 123 other than that which I have initiated... So they are hardly keeping customers in the loop.
And the advice they gave to rebuild the machines fell on its arse when I did it, got the email to say the new server was up and ready... But can't connect... And support are not answering.