back to article Tachyon Nexus theorises on ultra fast storage

Any in-memory-focussed clustered system has to deal with failure at some point, and learn how to recover from, or tolerate, it. Replication is a common method but it slows down processing, especially in sequences of jobs in a pipeline. However, upstart Tachyon Nexus thinks it has found a way round that problem, and can go a …

  1. Ian Michael Gumby Silver badge
    Boffin

    Meh.

    While Tachyon is interesting. It can't stand alone.

    You will need to persist data for one simple reason.

    You can only fit so much data in to memory before you eventually have to deal with swap.

    1. Anonymous Coward
      Anonymous Coward

      Re: Meh.

      Indeed. Were this a couple of years ago when Hadoop workloads were almost always IO bound this would be tremendously interesting, but Spark has put the brakes on that little train. Spark takes the same lineage-based in-memory strategy during a job that Tachyon does for storage, but works over HDFS (and half a dozen other file systems), and will happily run on-disk, so the benefits of an in memory filesystem during a process will be minimal.

      This was borne out in an excellent paper released this week demonstrating that Spark workloads have pushed the bottleneck to where it belongs: into the CPU [ http://www.eecs.berkeley.edu/~keo/publications/nsdi15-final147.pdf ]. They demonstrated that across the benchmarks used (which are representative of the vast majority of real-world work), eliminating the disk would improve performance by 19% at most.

      So yeah, 100x throughput is nice, but in reality it's going to be a very niche application - the only time I can really see this being of benefit is if you've got a large graph of jobs, requiring a write back to disk to share the data between them, but that's a rare use case in my experience and when it does happen isn't so latency-sensitive as to need the improved throughput.

      1. Ian Michael Gumby Silver badge

        Re: Meh.

        The issue is that you would want to use Tachyon and Spark in conjunction with a distributed file system.

        MapRFS, IBM's... even Cleversafe's.

        Things are getting interesting. And even Tachyon may become moot with RRAM although if its better writtent than Hadoop components... it could still be a winner.

  2. CaitlinBestler

    Replication is the only protection from network splits

    A journal can only help you recover if your network is intact.

    If this scheme only leaves a single journal and a single life copy of the new data I don't see how it can be robust during network failures.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019