back to article Why data storage technology is pretty much PERFECT

Reliable data storage is central to IT and therefore to modern life. We take it for granted, but what lies beneath? Digital video guru and IT author John Watkinson gets into the details of how it works together and serves us today, as well as what might happen in the future. Brain cells at the ready? This is gonna hurt. …

COMMENTS

This topic is closed for new posts.
  1. Charles 9

    "*It's violet"

    Cute. An advertising falsehood that actually UNDERSTATES the reality. That's something you don't see everyday.

  2. AceRimmer

    "No more powerful code can ever be devised and further research is pointless."

    Having committed that to print the Law of Sod dictates that you will be proved wrong

    Possibly by the quantum computational devices currently being devised.

    1. Mage Silver badge
      Megaphone

      Law of Sod dictates ...

      We'd need to discover some fundamental under pinning of Physics and Mathematics is totally flawed:

      Laws of thermodynamics in simple English:

      You can't win

      You can only get close to breaking even sometimes.

      you can't even leave the game

      Shannon - Nyquist:

      For any particular communications system (transmission, storage) you have a finite capacity that can't be exceeded. We get pretty close to that on Storage and Wireless. Ultimately this is based on the three laws of Thermodynamics. Speed / capacity (transmission/storage), power, noise, bandwidth all related. On storage the physical size sets the power noise and bandwidth equivalents of Transmission systems to limit your capacity.

      Entropy is a bugger.

      1. been there, didn't like it
        Devil

        Re: Law of Sod dictates ...

        No need to wait. Turbocodes. 1993. French though, so possibly doesn't count.

        1. This post has been deleted by its author

  3. Alister

    A fascinating and informative article, thank you!

  4. harmjschoonhoven
    Facepalm

    Not so random errors

    In the 1990s the Dutch retail chain "Kruitvat" sold low-cost recordable CDs.

    When a colleague showed me the physical location of the sectors where error correction was necessary a few months after recording, I could read "Kruitvat", exactly matching the logo proudly printed on the CD in the wrong kind of ink.

  5. Ted Treen
    Thumb Up

    Thank you

    An excellent article - which is how the Reg used to be all the time.

    1. Anonymous Coward
      Anonymous Coward

      Re: Thank you

      I agree, that said an article about Windows 8 would be generating far more comments.

      I did notice the Win8 install DVD in the picture on the first page, I wondered if that was there to troll some comments out of the usual suspects?

      1. Adam 1

        Re: Thank you

        WHAT USUAL SUSPECTS? FAIL

  6. Hayden Clark Silver badge
    Boffin

    Error correction isn't good enough nowadays.

    If you look at the uncorrected error rates for modern hard drives, they seem very low (e.g., 1 bit in 1x10^14 bits). However, there's 8x3x10^12 bits on a 3TB drive, 2.4x10^13. So you can expect a bit to be wrong every 4 times you rewrite the drive.

    That doesn't seem very much to me.

    1. nerdbert
      Holmes

      Re: Error correction isn't good enough nowadays.

      Bwahahahaha! You don't know the *half* of it.

      Let's take the example of a typical disk drive. In the bad old days, we had "peak detector" systems where bit densities were less than 1.0 CBD - 1 bit in a half pulse width. With enough work and some really simple coding you could reach about 1e-9 BER.

      Then IBM came up with the idea of applying signal processing to the disk drive, introducing partial response/maximum likelihood systems (Viterbi detectors) where you started to get more than 1 bit in a pulse width and the raw BER off the disk started to drop. Now they're putting about 3 bits in a single bit width because they're putting in LDPC codes and their 6M+ gate decoders behind the PRML and the raw BER coming off the disk is typically around 1e-5, but with the coding behind them they're typically well below 1e-15.

      You want scary? Look at MLC NAND flash drives. After a few hundred erasure cycles the raw BER of those things can be 1e-4 or worse. Why? Feature sizes are getting so small that leakage and wear (threshold voltage shifts, etc) are causing those ideal voltage levels to get pretty whacked out. It's getting bad enough that you're starting to see those massively complicated LDPC codes in flash drives, too. Those fancy codes are needed, as are wear leveling, compression, and all those other tricks to make NAND drives last as long as they do.

      HDD systems typically fail from mechanical failures but the underlying data is maintained and you can usually get someone to haul the data off the platters for enough money. NAND flash systems, though, die a horrible death from aging and if you have a "crash" on one of those it's not likely that any amount of money will get your data off it because of all the massaging of the data we do to keep those drives alive.

      1. Nigel 11

        Re: Error correction isn't good enough nowadays.

        HDD systems typically fail from mechanical failures but the underlying data is maintained and you can usually get someone to haul the data off the platters for enough money.

        If "spend enough money" is not an option, always give ddrescue a chance! In my experience many drives fail soft: they report so many errors to an operating system that they look dead, but a utility that tries and tries over and over again in an intelligent way (as ddrescue does) will eventually retrieve all the data, or all but a very small fraction of the data.

        Of course, you need backup, for the times that the drive does turn into a brick, just like that.

        Also always keep an eye on the drives' SMART counters. In my experience many failures are flagged a long time in advance by an exponentially increasing number of reallocated blocks. I pre-emptively replace such drives, long before the error count hits the SMART failure threshold. It's the rate of increase that's the give-away, not the absolute number. I have one drive with a few hundred reallocated blocks from the first time it was written to, but that number hasn't increased by as much as one since then.

    2. Anonymous Coward
      Anonymous Coward

      Re: Error correction isn't good enough nowadays.

      Also, storage media have initial manufacturing faults that can result in local data loss in excess of what error correction can restore (although it can easily detect it). The usual solution is to map those faults out once they've been detected. But the first time you write the medium and then read it back, you'll probably have a few non-recoverable errors because they're only detected on the first read. With media being so big these days, the manufacturers don't have time to do that bit for you.

      So if you really don't ever want to lose data, you have to write and read the entire medium before you use it for real.

      1. nerdbert
        Holmes

        Re: Error correction isn't good enough nowadays.

        Nope, you don't need to do that, or at least it's very, very rare to have problems with that.

        What we do these days is that we can detect errors and weak sectors using various intermediate code output stages to estimate the SNR of the read (think SOVA systems and the like). If we detect a bad or weak readback sector while reading we map out the offending block and use a spare one in its place. It's completely transparent to the user and it keeps us from having to wear out NAND any more than is absolutely necessary. (Something similar is done for HDDs.) You have to have a complete failure before something like this causes a problem that's visible to the user.

        But think about what this means to end users. it means that if you ever start getting bad sector warnings what's happened is that we've used up all our spares and that we can't safely remap bad sectors without OS level help. That means that your storage device is on its last legs and you'd best be getting anything valuable off the drive ASAP since the aging doesn't ever stop.

        1. Anonymous Coward
          Anonymous Coward

          Re: Error correction isn't good enough nowadays.

          nerdbert: "You have to have a complete failure before something like this causes a problem that's visible to the user."

          Yep, that's the sort of complete failure I was referring to; failure to read any data after the first write. I find it to be rather more common on new hard disks than you suggest.

      2. Nigel 11
        Thumb Up

        Re: Error correction isn't good enough nowadays.

        So if you really don't ever want to lose data, you have to write and read the entire medium before you use it for real.

        Amen to that.

    3. Nigel 11

      Re: Error correction isn't good enough nowadays.

      This is why you put extra checksums in things that really matter.

      ZFS, for example, does end-to-end checksumming, so it'll spot when a disk drive has corrupted data, and recover using redundantly recorded data where that is available (always, for metadata). Because the ZFS checksums are done at the CPU, it'll also spot errors between the disk drives and the CPU ( a failing disk drive controller, a flaky memory controller).

      If your filestore can't be relied upon to protect you, store SHA256 sums of the data in your files. You can't correct using an SHA256, but the chances of a data error getting past it is truly infinitessimal.

  7. Rol

    Make those bits work harder?

    While the states of polarity lend themselves to binary storage quite readily, I am thinking the intensity of that polarisation could be the next leap forward.

    Hard drives and indeed all magnetic media, flip many particles polarity to record a single bit, so not all are aligned in one direction, but enough to represent a 1 or 0

    If by finely adjusting the intensity of a write, I can finely adjust how many particles flip, I can store more data in that discrete location, without necessarily having to work at a smaller scale.

    Effectively I could be writing hexadecimal digits in locations that historically stored only a binary bit, just by adjusting the intensity of the write by a step of 8.

    The leap will require ever more precise power management and the state of the bit before write would also need to be known, but with the authors assertions of error corrections ability and the advances made since magnetic storage came on the scene, I see this being achievable.

    My 2TB drive would then become a 32TB drive, without the need to pack the data closer, just the ability to make finer measurements than N or S.

    I know someone has the ability to poke enough holes in this to make it sink, but I wonder who will get there first.

    Ready.... steady..... go!

    1. Charles 9

      Re: Make those bits work harder?

      The thing with your idea is that if you can alter polarities more precisely, it would be easier to just use smaller areas to represent the bits, which is what' has been happening steadily with hard drive technology: the surface area of the drive used as the "bit" gets smaller and smaller. Trouble is, we're reaching a point where physics really is getting in the way. Not in the sense that we're down to flipping single molecules but still at the point where the latent magnetic fields of adjacent "bits" could cause the bit you're on to naturally "rot" and spontaneously switch. That's why the current push is for HAMR: Heat-Assisted Magnetic Recording. If the bit can't be changed unless above a certaintemperature, then it's less prone to neighboring fields and can be made even smaller.

      1. Rol

        Re: Make those bits work harder?

        As you quite rightly pointed out, the data is being packed so close together it is bleeding over into adjacent cells, whereas my solution is to take the space one bit would occupy or perhaps more conveniently the space it occupied several years ago before neighbour contamination became an issue and make that space represent any of 8 different intensities of 2 polarity states.

        Cell density would be at late 20th century levels, but each cell could hold 2 bytes of data instead of 1 bit.

        I guess a method to keep the data refreshed over time, beside the use of a serious error correction algorithm could work.

        1. Toastan Buttar
          FAIL

          Re: Make those bits work harder?

          In order to get 16 times the storage capacity, you would have to 'write' and 'read' 65536 discrete bands of 'intensity', not 8.

          3-bits-per-cell NAND memory encode 8 bands of voltage to store those three bits. You require 2^n detectable levels to store n bits.

          1. Toastan Buttar

            Re: Make those bits work harder?

            Slight correction: If you used +/- 8 levels of analogue 'intensity', you could store 4 bits. My bad.

        2. Charles 9

          Re: Make those bits work harder?

          That's assuming you can accurately read, write, and maintain 2^n different strengths of magnetism less expensively than just partitioning the space into n cells. The readers may be tiny, but they only have to detect on/off, which is a whole lot simpler than a reader that has to distinguish a spread of different strengths. Also consider, these larger cells may be larger but also more sensitive to changes because there are more ways a change can be registered, meaning neighboring fields may be MORE likely to introduce errors, not LESS.

          1. Rol

            Re: Make those bits work harder?

            I think I need a bigger boat!

            The wholes in this one are beyond any correction algorithm I know.

            1. Rol

              Re: Make those bits work harder?

              ..and thanks for the education

    2. the spectacularly refined chap

      Re: Make those bits work harder?

      Hard drives and indeed all magnetic media, flip many particles polarity to record a single bit, so not all are aligned in one direction, but enough to represent a 1 or 0

      If by finely adjusting the intensity of a write, I can finely adjust how many particles flip, I can store more data in that discrete location, without necessarily having to work at a smaller scale.

      If you have a slack hour or two sometime sit down and read up about how magnetic core memory operates. Sure, magnetic media has moved on a lot since core but it is still the same basic principles at play and somehow people seem to grasp magnetising ferrite beads easier than patches on a tape or disk. Magnetic domains essentially have two states and that is the lot - core memory exploits that to the full. The basic idea was that there was a specific field necessary to polarise the magnetism in a core in one particular direction. Once that had been done you could send as many lesser field strength pulses through it (either the same way or the opposite way) as you like without altering the state of the core in the slightest.

      This was done in normal operation so that whole rows and columns of cores could be addressed with a single wire. It was only when the current in two or more wires added up to meet that threshold again that the core "flopped" into the opposing state when again it was equally stable. Two states → binary system. Analog recording is very much working against that fundamental property of the media, not in harmony with it.

      1. Nigel 11
        Boffin

        Re: Make those bits work harder?

        Analog recording is very much working against that fundamental property of the media, not in harmony with it.

        To the extent that analogue tape recorders are in fact very crude digital recorders. The record head is fed with an ultrasonic signal that generates a stream of 1 and 0 bits on the tape. The Analog signal modulates this signal so that the 1 bits become wider or narrower on the tape compared to the 0 bits. The read head averages out the magnetisation of a few "bits" worth of tape, to retrieve the audio. A filter in the subsequent amplifier removes what little remains of the ultrasonic carrier after averaging.

        The very earliest magnetic tape recorders didn't use the ultrasonic carrier. The technique was discovered accidentally, whem a component failure caused a parasitic oscillation to develop in the recording amplifier, and the fidelity of the recording became spectacularly better as a result!

    3. Jim Bob
      Alien

      Re: Make those bits work harder?

      "While the states of polarity lend themselves to binary storage quite readily, I am thinking the intensity of that polarisation could be the next leap forward."

      Polarity is a binary property, intensity is an analogue one.

      So you are suggesting that one takes a digital system and discard the simple (cheap) reader/writers and easy error correction.

      Big enough hole for you ;o)

  8. DavCrav

    "In contrast, anyone can understand water flowing in a pipe and that is why our drinking water system is in such a shambles - with much of it running to waste through leaks."

    I attended a lecture by a civil engineer a few days ago where he said that without the copious leaks in our water system, all the trees in city streets would die. So it's a trade off.

  9. Steven Jones

    hmm...

    "In contrast, anyone can understand water flowing in a pipe and that is why our drinking water system is in such a shambles - with much of it running to waste through leaks."

    Simply speaking, that's complete nonsense. There's no reason leakage in drinking water systems need to achieve anything remotely like losses of one in many orders of magnitude. It's simply a cost tradeoff. It's perfectly feasible to achieve very low levels of loss in systems using fluids (how many people have refrigerators or freezers running happily after a couple of decades - as mine does), but it makes no sense at all to do the same with water delivery where (depending on the costs and availability of collection vs demand) less rigorous standards of loss are required. It also makes no financial sense to use expensive engineering, scientific and mathematical specialists on such a requirement.

    Quite simply, a completely pointless - and indeed, misleading comparison. The reason high water loss levels are tolerated is nothing at all to do with the easy understanding of the technical issues, and everything to do with the costs involved. This simply reads as a smug pop at another discipline for no good reason as the criteria of success are wildly different.

    Incidentally, no error correction is "perfect". All have tolerance levels - in fact, as others have pointed out, the theoretical error levels which can be achieved on disk drives are such that,on very large, and very active storage systems, uncorrected (or, much worse, undetected errors) are a realistic possibility. (And not just storage systems - data transmission, electrical interfaces and others can suffer this way). Not to say that the theoreticians, engineers and solid state physicists who have achieved this haven't done something approaching the miraculous, but perfect it isn't. Indeed it's an impossibility

  10. Bob Merkin
    Trollface

    Green beans?

    I didn't know frozen green beans were a digital storage medium. If they thaw out, do they lose their data?

    Oh . . . the bar code. Gotcha.

  11. Anonymous Coward
    Anonymous Coward

    No more powerful code can ever be devised; further research is pointless

    Many thanks for linking to the original article. It's the first time I'd ever read it, and it's refreshingly brief and understandable from a mathematical standpoint, even if you know nothing about error-correcting codes. I was originally intending to try to find fault with the "further research is pointless" comment by trying to find a shortcoming in the paper itself, but after careful re-reading all my objections vanished. Instead, I'd just like to point out that if there had been no further research done because "no more powerful code can ever be devised" then we wouldn't have had the theory re-evaluated to work as a means of securely sharing secrets under the guise of Michael Rabin's Information Dispersal Algorithm. Nor would we have the less mathematically perfect, but nonetheless much more efficient to calculate, attempts at error-correcting schemes such as Turbo Codes or using Bayesian belief-propagation networks.

    In the end, I can argue (however weakly) with the "further research is pointless" quip, but not with the other assertion that nothing more (mathematically) perfect can be devised in this realm.

    1. Charles 9

      Re: No more powerful code can ever be devised; further research is pointless

      I suspect the "further research is pointless" merely points to the fact that we won't be able to find a better error-correcting system overall than Reed-Solomon because they account for physical limitations. You can create something of equal performance, but never better performance. As you've noted, research instead has turned more toward adaptation (the Information Dispersal Algorithm) and specialization (Turbocodes). I sometimes wonder if someone can find a system equal in performance to Reed-Solomon but simpler to implement, but I'm not an expert in those matters.

      1. DavCrav

        Re: No more powerful code can ever be devised; further research is pointless

        Reed--Solomon is close to the theoretical limit given by the sphere-packing bound, a so-called "perfect" code. There are trivial codes, Hamming and Golay codes, and that's it for perfect codes. Reed--Solomon is close to perfect, and so there is very little else you can do. However, that isn't to say that you cannot do better in terms of finding codes that are easier to implement, or slightly closer to the sphere-packing bound.

        Reed--Solomon is optimal in the sense that it satisfies the Singleton bound, a bound that only applies to certain codes, called linear ones. If your code is non-linear, then you will need a look-up table, which might be quicker or slower, depending on the size of the code. (Of course, CDs are not encoded using standard RS, but striped across various parts of the disc then re-encoded using a look-up table designed to get rid of short and long pits and lands, as mentioned in the article.)

        So in some sense there is nothing more that can be done, and RS is the best possible. However, you can get away with less error correction if you think your reliability is better.

  12. Jim 59

    Hamming

    Why so little credit to Richard Hamming ? He gets barely a single mention, but was seemingly was the granddaddy of error correction ?

This topic is closed for new posts.