Where has all the bad storage gone?

@Paul Bruneau

Yes, there were once amos32 users. I quite liked it, actually. Except for vcrsav & vcrres, of course. And then there was dskpak, which always started yet another pass just when you thought (or hoped) the damn thing had finished. The rdbms (IMS) was pretty good, though.

@Gaius - re rsync

Rsync does not by default overwrite. It writes new, then deletes (unlinks) old.

Pedantry, one might think. But not so. In fact, Linux offers the means to roll your own time machine. The key to this is cp -al to create a hardlink farm duplicating the current state of your backup, then rsync to update the hardlink farm. Files that have changed will be updated, but the original will still be accessible through the original link, and won't get deleted until its link count reaches zero.

See http://www.mikerubel.org/computers/rsync_snapshots/ for a good explanation.

As to why no Windows time machine? I'd guess that NTFS is not a linux-like FS with inodes and hardlinks, which therefore can't easily support this sort of magic. Not that I'd regard "backup" to the same hard disk, or even the same system, as what I'm backing up from, as secure. The backup disk needs to be far enough away that it won't be stolen by the same thief, nor destroyed by the same flood or fire.

I'd bet my lunch...

...that Apple uses the techniques described here:

http://www.mikerubel.org/computers/rsync_snapshots/

I've used a variation of this on Linux for years, but it should be OK on Windows (there is a section in the FAQ about some cygwin rsync issues, but there are workarounds). This has saved my bacon several times, as I have suffered from more than my fair share of hard-drive failures (actually the reason I read this misleadingly titled article).

How it works is fully described if you follow the link, but in summary, each backup stores a complete snapshot of the filesystem, but unchanged files are simply linked. If you're canny, you can have daily backups, with hourly snapshots since the last daily, and 5-minute backups since the last hourly. You therefore need never lose more than 5 minute's work.

For typical usage patterns, where most of a disk is unchanged, a backup drive about twice the size of the live drive can store backups going back several years, even if snapshots are taken every 5 minutes. All snapshots can be accessed just like a live filesystem. Comparing two versions of a file is simplicity itself, and file recovery is just a simple file copy.

Of course, there's no slick GUI, but once it's up and running, there's no need for any UI at all - it just works, and because the backups are live filesystems, you can access files in the snapshots using whatever software you want.

Replication is not backup either!

Ditto on the comments about raid, but simply using rsync or other techniques to copy your files to a remote drive isn't a backup strategy either.

What happens when the file you just copied turns out to be corrupt? You need an older version.

How many older versions do you need? In one case where a telco killed their system, they had to go back 14 MONTHS to find data images for a phone exchange that weren't corrupted, then wind in all the incremental changes over a 4 week period. Meantime 90,000 people had dodgy phone service (I was one of them)

Home backups are a pain in the a***, primarily because of media fragility and where to keep it safely (backing up 1Tb+ onto DVDs is a no-go, big tapes are pricey and hard drives are touchy)

For $orkplace I use Bacula, but I'm backing up several tens of Terabytes with it and have all the right gear (tape robot, fibre, giant data safe, blah blah etc ) to go with it.

Bacula's fairly simple to setup for home use. It doesn't have a whizzy gui and you need a propellor beanie to set it up, BUT it works really well.

The online services seem like a good idea, modulo the question of "can you trust them?" - encypting your data BEFORE it goes upstream might solve that though - but you still have the issue of blowing your data cap every time you make a full backup. :(

Mine's the one with the tinfoil beanie in the shoulder pocket.

Tar cvf ... ...

mine's the one with the unix manual in the pocket.

Err... pockets.

Why Time Machine works

Time Machine takes away the constant battle of effort trying to decide what needs backing up and how to back it up - it turns your disk from being a hard-drive with files and directories into more like the idea non-techies have of what that disk is, a place that stores everything you do and allows you to go back to old versions simply if you need to. That it also helps when the hardware is upgraded, lost or broken is even better.

I'm sure all the readers are aware that the weakest link in any backup strategy is the fleshy button-prodder. Apple have looked at the problem in terms of people who are used to the browser recording every site they've visited and the word-processor remembering every change they made to a document (more or less, depending on things like whether the program was quit out of or not) (anybody remember the apocryphal "some users" tale of the man with one huge word doc of every document he'd ever written?) but somehow when things are written to disk (or "memory" depending on the level of understanding) they can get corrupted, lost etc.

I like Apple's design strategy for these things - I've always used various unixes (including inevitably now some godawful linux) at work, when I go home I want to use a computer that's taken care of all the things I have to worry about in the office. I don't want another job at home, I want a computer that has tried to save me from worrying about them. Adding backups and transfers to the list of these things that have been thought out and essentially just work is as good to me as when the iPod worked out that spinning a wheel was a great way to browse through my albums along with the breakthrough that software I used to play mp3s on the computer would also be the software that would copy them to the iPod.

Backing up isn't hard when it's a job or a hobby ... but when you just want it to work while living in blissful ignorance (I can't make the car engine metaphor here because it's overused and truthfully I do have more understanding of how they work than is helpful in most situations, but normally this is where that cliche creeps in) it's easier to ignore it and hope for the best than try to weigh up the options, formulate a plan (do I need to backup weekly, daily, monthly ?!) and remember how to get stuff back from backup, how to search and find the version / file I want to restore and many other things that Time Machine has essentially solved.

To backup, plug in backup disk. To backup continually leave disk attached to computer. To restore (while backup disk attached - or capsule is switched on, for the wireless version), find/search document then hit Time Machine button to go back in time to before you broke it. Would you rather explain this to your parents or how to use rsync ? (Assuming you could get them over the shock of saying "arse-ing" so very many times.)

wwphd?

Storage?

I was assuming this was going to be an article about what disks to avoid/ the history of how they have become more reliable. We got a bitch about tape storage and then an advert for Apple.

Whatever.