Fasthosts admits email destruction fiasco

Plusnet...

The bloke whose finger trouble caused Plusnet's email disaster must've moved to Gloucester. Boom,boom! (in more ways than one)

Mail backups

"I guess they just make a mirror, and don't archive snapshots or do incrementals"

Do you *really* want your ISP keeping incremental backups of all of the mail they've ever received for you?

Your ISP might never need those backups, but it's a slippery slope from there to the government requiring your ISP to not only keep those backups but to hand them over on demand too...

Some people never learn!

It was not all that long ago that I received an email from those lovely people at Fasthosts informing me that whilt carrying out routine maintenance work overnight, they had inadvertantly wiped the hard disks of a number of servers. Regretably, apparently, they were unable to provide any backups of lost data but they were looking to ensure that this could never happen again.

Seems they forgot.

A single copy is no backup

Any "backup" process that can corrupt all backup copies when an error occurs doesn't deserve the name. You need at least two generations of backup to guarantee data safety. There must be at least one acceptably recent backup generation offline, and preferably offsite, at all times.

In this case the backup process succeeded, but with damaged data being saved which made the backup worthless. However a power outage, equipment failure or fire during the backup could have destroyed both the main disks and the backup medium. Its the same result in both scenarios: without an earlier offline copy you don't have anything to fall back on.

Fasthosts evidently got caught by the first case but could just as easily been hit by the second.

I must have annoyed someone in a previous life..

I'm both a Force9/Plusnet AND a Fasthosts customer, now I just need Donhost to go deleting stuff on me, and I'm 3 for 3....

Re: Some people never learn!

@ Matt

Chances are the e-mail to remind themselves not to lose data never arrived or their server 'accidently' deleted it LOL

Proper Backups...

Email should be on a 7 day rotating backup. That should be enough time to notice an error like this and recover, but not so long as to be usefull to government snoops. Anyway, can't they send the disk out for STM data recovery? Seems like this might be one of those times when it is justified.

comedy of errors

so this must be that period last week when none of my emails were responded too, and the ones that i sent TO MYSELF never arrived. marvellous fasthosts. actually some of them did arrive 8 hours later. excellent.

thank you reg for giving me an explanation that was not forthcoming from fasthosts themselves.

it did actually force me to create a googlemail account, which until now i had been reluctant to do on the grounds that they store everything for government processing for ever.

hmmm. ok. well, just leaves me with one gripe, which is to find out how someone accidentally created a mysql database in my account. i trawled through the mystery database emailed the administrator contact and it turned out that they had just been trying to create a database in their own account. ridiculous.

are the humans at fasthosts more prone to errors than in other hosting companies?

oh, and the frustration of them merging several of my accounts into one earlier in the year and changing my username. this would be fine except that 3 other people

shared access to one of those accounts which was very deliberately intended to be kept seperate from my personal one.

fasthosts is a complete joke a comedy of errors.

a mess.

a nonsense.

a laughing stock.

Keep it simple stupid....

You should of course run a backup BEFORE doing any major upgrades/housekeeping not AFTER, then ensure it has worked before starting your work (as much as is possible). Belt and braces says you run one after as well and these are kept out of the standard backup cycle for a period until you are sure there are no negative affects/data loss.

Any upgrade/housekeeping 'plan' should have a regression plan and a testing phase. Clearly there was neither. Questions should be asked of who ever is in charge and signed the work off. Simple change control procedures and standard operating procedures prevent cockups.

Pretty simple stuff.

Where's my spam?

I was wondering where all my spam went and why my mailbox was empty.

And the good news is...

"The firm apologised and says its fix means the same disaster cannot happen again."

I'm sure that FH customers will be ecstatic to know that FH will be bringing them *new* disasters, rather than simply recycling the old ones.

This is a perfect example of why I run my own servers.

Trust no one. Keep your laser handy. The Computer is your friend.

Rubbish MTA's????

@Peter Knapp

Since when was Linux and a decent open source MTA be classed as rubbish? You can have a fully redundant and reliable MTA platform setup relatively easy if you know what you're doing.

I'm just guessing the guys at FH were simply lacking in the "know what you're doing" section...

Can I just say ...

... that those people in the "wot no backup" and "inadequate backup" camps should try actually doing it. It's trivial to do a backup of someones desktop - kick them off, do a backup, job done.

Now try doing a backup where files are appearing and disappearing at a startling rate - at a place the size of FH, probably thousands a minute. Yes, ANY snapshot backup you take is out of date and incomplete (not to mention inconsistent) BEFORE it's finished. Sure you can do real-time stuff with a mirrored filesystem - but then when you type the wrong command it takes out the clone at the same time !

The only way I can see to avoid this sort of problem would be to duplicate all emails as they come in and keep a separate backup store - then if you lose the mail you can go to the backup store and re-inject all the mails for the last <however long it's been>. That would be a huge store with it's own reliability problems, and you'd still have the problem of having to go back 3 (or more) weeks for the person that never checks their mail whilst "somewhat annoying" everyone else who suddenly gets all their mail (and spam) from the last three weeks for a second time.

If anyone SERIOUSLY thinks they have an answer then I'd like to hear it - whining pillocks who think it's as simple as managing two desktops in a network need not apply !