Fasthosts, "the UK's number 1 web host", has admitted to a botched update to its mail server that permanently deleted customer emails. The cockup last week was compounded when the firm's backup system failed, meaning about half of the affected emails are lost forever. Fasthost sent us its side of the story: Following a human …
Doesn't Sound Like Much of a Backup
I guess they just make a mirror, and don't archive snapshots or do incrementals.
That's the problem with backups. They are expensive as hell, no one likes thinking about them, they should never be needed, and you always find out they're broken when you need them most.
When they say "50% of the emails..."
So, all the spam about 'extensions' and 'tablets' have been restored, but that email from your relative in Australia...
The bloke whose finger trouble caused Plusnet's email disaster must've moved to Gloucester. Boom,boom! (in more ways than one)
"I guess they just make a mirror, and don't archive snapshots or do incrementals"
Do you *really* want your ISP keeping incremental backups of all of the mail they've ever received for you?
Your ISP might never need those backups, but it's a slippery slope from there to the government requiring your ISP to not only keep those backups but to hand them over on demand too...
Yes, the grounds for Fasthosts being the number one provider seem pretty thin to me too. Could find anything on the netcraft site that they refer to endorsing them. And what's so could about being the biggest anyway..?
Some people never learn!
It was not all that long ago that I received an email from those lovely people at Fasthosts informing me that whilt carrying out routine maintenance work overnight, they had inadvertantly wiped the hard disks of a number of servers. Regretably, apparently, they were unable to provide any backups of lost data but they were looking to ensure that this could never happen again.
Seems they forgot.
Why use in house email?
I remeber seeing an article just the other day, might have been Gartner, not sure, saying that in house email is expensive and many companies should just use hosting services... Looks as if we can see why the hosting services are cheap, doesn't it. Put the backups to /dev/null and they complete in no time every day...
A single copy is no backup
Any "backup" process that can corrupt all backup copies when an error occurs doesn't deserve the name. You need at least two generations of backup to guarantee data safety. There must be at least one acceptably recent backup generation offline, and preferably offsite, at all times.
In this case the backup process succeeded, but with damaged data being saved which made the backup worthless. However a power outage, equipment failure or fire during the backup could have destroyed both the main disks and the backup medium. Its the same result in both scenarios: without an earlier offline copy you don't have anything to fall back on.
Fasthosts evidently got caught by the first case but could just as easily been hit by the second.
I must have annoyed someone in a previous life..
I'm both a Force9/Plusnet AND a Fasthosts customer, now I just need Donhost to go deleting stuff on me, and I'm 3 for 3....
Re: Some people never learn!
Chances are the e-mail to remind themselves not to lose data never arrived or their server 'accidently' deleted it LOL
As an Ex-Support @ FH...
I know that their backup procedures are second to none... Which is actually close to the truth.
Any mention of the words 'back' and 'up' within those not so hallowed corridors were met with sniggers, and not just from the Support Staff.
Email should be on a 7 day rotating backup. That should be enough time to notice an error like this and recover, but not so long as to be usefull to government snoops. Anyway, can't they send the disk out for STM data recovery? Seems like this might be one of those times when it is justified.
comedy of errors
so this must be that period last week when none of my emails were responded too, and the ones that i sent TO MYSELF never arrived. marvellous fasthosts. actually some of them did arrive 8 hours later. excellent.
thank you reg for giving me an explanation that was not forthcoming from fasthosts themselves.
it did actually force me to create a googlemail account, which until now i had been reluctant to do on the grounds that they store everything for government processing for ever.
hmmm. ok. well, just leaves me with one gripe, which is to find out how someone accidentally created a mysql database in my account. i trawled through the mystery database emailed the administrator contact and it turned out that they had just been trying to create a database in their own account. ridiculous.
are the humans at fasthosts more prone to errors than in other hosting companies?
oh, and the frustration of them merging several of my accounts into one earlier in the year and changing my username. this would be fine except that 3 other people
shared access to one of those accounts which was very deliberately intended to be kept seperate from my personal one.
fasthosts is a complete joke a comedy of errors.
a laughing stock.
You had to pay extra for this to happen!
FFS, They actually had email to loose!
From our long standing time co-lo'ing and previous to that hosting with FH I am quite impressed they actually had email to loose. The 50% they lost was probably about half of the 50% they actually deliver and don't "loose" in the MTAs along the way. This applies to both Exchange and all their lesser mail boxes as they insist on using the same rubbish MTA's for all of them (a shelf of beige box servers with linux on them).
At least their customers can now say they only lost 25% of their email due to this fault as the rest was dumped at the MTA due to disk and processor contention.
I am pleased to say the customers we took with us happily receive two disk snapshots per 6 hours in two different locations, held until the NAS's are full, then oldest in is deleted first. We are at about 8 months retention at the moment, so bring it on FH!
Keep it simple stupid....
You should of course run a backup BEFORE doing any major upgrades/housekeeping not AFTER, then ensure it has worked before starting your work (as much as is possible). Belt and braces says you run one after as well and these are kept out of the standard backup cycle for a period until you are sure there are no negative affects/data loss.
Any upgrade/housekeeping 'plan' should have a regression plan and a testing phase. Clearly there was neither. Questions should be asked of who ever is in charge and signed the work off. Simple change control procedures and standard operating procedures prevent cockups.
Pretty simple stuff.
Where's my spam?
I was wondering where all my spam went and why my mailbox was empty.
Further impressed that FH now (presumably) have a "load balanced" MX cluster
Hiding behind their equally weighted AS Peers, which explains their transit flips 24x7...
Or is it not load balanced and just a single host!
OMG, you folks really need to look elsewhere ;0
fasthosts.co.uk MX preference = 5, mail exchanger = mailserver.fasthosts.co.uk
mailserver.fasthosts.co.uk internet address = 18.104.22.168
And the good news is...
"The firm apologised and says its fix means the same disaster cannot happen again."
I'm sure that FH customers will be ecstatic to know that FH will be bringing them *new* disasters, rather than simply recycling the old ones.
This is a perfect example of why I run my own servers.
Trust no one. Keep your laser handy. The Computer is your friend.
now we know
Why they are so fast running from murderous customers. Seriously this company must really suck I wouldn't host a parking page with them.
Rest easy on issues of privacy!
I never fail to be amused by people who worry that the government might be siphoning of the emails and/or web searches of SOME people SOME of the time ...
Since when was Linux and a decent open source MTA be classed as rubbish? You can have a fully redundant and reliable MTA platform setup relatively easy if you know what you're doing.
I'm just guessing the guys at FH were simply lacking in the "know what you're doing" section...
Can I just say ...
... that those people in the "wot no backup" and "inadequate backup" camps should try actually doing it. It's trivial to do a backup of someones desktop - kick them off, do a backup, job done.
Now try doing a backup where files are appearing and disappearing at a startling rate - at a place the size of FH, probably thousands a minute. Yes, ANY snapshot backup you take is out of date and incomplete (not to mention inconsistent) BEFORE it's finished. Sure you can do real-time stuff with a mirrored filesystem - but then when you type the wrong command it takes out the clone at the same time !
The only way I can see to avoid this sort of problem would be to duplicate all emails as they come in and keep a separate backup store - then if you lose the mail you can go to the backup store and re-inject all the mails for the last <however long it's been>. That would be a huge store with it's own reliability problems, and you'd still have the problem of having to go back 3 (or more) weeks for the person that never checks their mail whilst "somewhat annoying" everyone else who suddenly gets all their mail (and spam) from the last three weeks for a second time.
If anyone SERIOUSLY thinks they have an answer then I'd like to hear it - whining pillocks who think it's as simple as managing two desktops in a network need not apply !
Well it doesn't surprise me things are going "belly up" for Fasthosts - things are getting very bad for them - any one can see that.
Backup systems failing
Internal Databases hosted on NT4 (old) hacked etc
Whats next ?
I for am glad I dont host any sites on there systems. I found a far superior small UK host and they provide me with everythign I need - the personal touch, the time and the effort the chat about anythying
Personally - we wont be using FH for anything else - I just can't trust them with my domains - ill be paying £200 to transfer all my domains out
As per usual FH tell us no real details - hide the fact that their internal systems are insecure
new host are: www.sar-hosting.co.uk
- Analysis Oh no, Joe: WinPhone users already griping over 8.1 mega-update
- Leaked pics show EMBIGGENED iPhone 6 screen
- Opportunity selfie: Martian winds have given the spunky ol' rover a spring cleaning
- OK, we get the message, Microsoft: Windows Defender splats 1000s of WinXP, Server 2k3 PCs
- Episode 4 BOFH: Oh DO tell us what you think. *CLICK*