Tape lives! The tape archive bit bucket is becoming bottomless

Re: Don't bother with the 'compressed'

A lot of stuff for long-term (i.e. tape) storage would be compressed before it go to the tape drive

I agree you won't get the claimed compression rate very often, but it depends on your requirements whether you would want to compress before taping. If you are really after long term storage, LTO compression is a good choice, because the standard requires that all manufacturer's drives can decompress the data. If you compress with some external tool, are you sure the one in common use will decompress your archive years after the people responsible for archiving it have retired?

I have written 1/2" tapes and read them back 30 years later. (Admittedly not LTO tapes, obviously). The operating system used to write them is no longer available.

I have managed tor recover most data from 20 year old DDS tapes. If I had used one of the compression options available with versions of tar around when I wrote them, I would have to spend days figuring out how to configure it to uncompress the data, assuming I could find a suitable version of zip, winzip, gzip, or bzip or whatever (I might have opted for whatever that compression thing was on CDC mainframes, if I was in the mood - who was to know Seymour Cray was going to leave? - or IBM Squoze - as recommended by my Mum!)

Re: I like the fact the founder is still the CEO

Michael Dell - Dell ?

Larry Ellison - Oracle ?

Re: Long term problem

Well that is a process problem. The fix is to start using TSM or Spectrum Protect as they like to call it these days and stop taking tapes out your library, just have two libraries with a copy of the data in both of them.

When you upgrade to new tape technology you just get your TSM servers to copy the data to the new tapes. Might take a few months, and you will probably have to once a week take some old tapes out and feed new ones into the library, but you then end up with everything on the new tapes. Rinse and repeat each time you upgrade to a new generation of tape.

My top tip is to keep a couple of the old technology tape drives in the library along with some tapes and dedicate them to TSM's DB backup. That way your DB backups never get blocked, which is a very good thing.

Re: Long term problem

Regularly re-read random tapes from the vault just to see?

You will notice unavailable tape drives relatively quickly.

It's like checking that all the plutonium is still in the basement vault.

Re: Long term problem

how does one ensure that the Software is also still capable....

Use old Unix (BSD) tar to write to the tapes. Not Star or gtar, or any other newfangled tar. The original one.

Choose a blocking factor and stick to it. The "default" setting is not very portable over time (used to be 20 card images of 80 characters, now mostly 20 of 512 byte blocks).

Set the Block Size to variable, because fixed block sizes appear not very portable at all with different manufacturers: does 512 mean blocks of 512 bytes or 512 blocks of 512 bytes or 4k bytes, or something else? (depends which day of the week it is, mostly).

Use open source software, and save a copy of the source where you can find it. (probably not on an 8" floppy).

Probably not, so it would be nice if LTO quotes in future are raw and let the user assume responsibility for optimizing its use through compression, deduplication, etc.

Having said that, I miss the Travan days. At least back then, tape drives were within consumer reach and provided us with at least some means of offloading cold data in the days when 1GB of data was a premium. A tape system accessible and affordable to the consumer in tiers of 2, 4, maybe 8 to 12TB for packrats would at least provide an alternative to external drives which can have reliability issues. At this stage, the only one within reach (and at a stretch) is rust-based RDX. Longevity doesn't even have to be so strong. Five, maybe ten years on the outside would be enough to handle a move between generations if need be.

Re: I have a new 1PB+ storage option.

No, not more compressible but don't forget that even though the algorithms are fairly simple like that used for say NTFS compression that was introduced in Windows NT 3.51 it could easily reduce the size by around 50% for easily compressible data (whereas WinZip or WinRAR would shrink it down to say an eighth of the original size) .

Newer tape drives will have better algorithms and even if they don't they will most likely have larger memory buffers and work on larger chunks of data and something like LZW compression even back then would have gotten better results if you used a larger workspace going from a 128 KB buffer to say a 1 MB one and the last time I checked these drives may have like one GB of on-board RAM so its not inconceivable that they can get this sort of compression. Obviously if they get fed like Mpeg-2 data or random numbers then compression will be zero.

Re: I have a new 1PB+ storage option.

Is the compression algorithm something like this. Processed in binary, remove all zeros as this is just wasted space and dedupe the ones. By my reckoning you have invented infinite compression. Well done, if you will excuse me now I need to go an oil my perpetual motion machine.

Re: Interesting how it has evolved...

Herby - I still have some tape rings.

Mount ring out or ring in.

Re: Price range

For a very rough estimate, Amazon will sell you a "1.5-3 TB" tape for about 20 quid which means that the price per TB is about half that of spinning rust from the same emporium.

(Obviously that's not the tape product being discussed here, but hopefully the unfairness of my comparison will provoke someone who actually knows about the subject.)