Aspera high speed file transfer: Let the cloud protocol wars begin

In two words: reliable multicast

Cloudy replication, fail-over and transparent migration of workloads and associated data sets are bloody expensive to implement in terms of available network bandwidth. The standard answer of adding more bandwidth is not cheap and the underlying networking technology doesn't scale in line with CPU, RAM and disk storage. Demands for bandwidth looks set to outstrip our ability to provide it (at reasonable cost) for the foreseeable future.

There is, in theory at least, a solution in sight: the use of rateless erasure codes over UDP multicast networks. Also known as fountain codes, these would allow data to be replicated to any number of servers with minimal (often around 3--5%, probabilistic) overheads. These systems improve over traditional forward error correction because they work for any network error rates (traditional FEC simply fails if the erasure rate is too high). These schemes make it much easier to eliminate or minimise the number of explicit ACK/NAK messages that need to be passed back to the sender, enabling them to scale out as well as scale up.

Unfortunately, the technology has its problems:

* patent encumbrance (some patents relating to LT, Raptor and fountain codes in general are overly broad)

* no hardware support (though this could change, for now everything has to be done in software)

* latency issues

Latency seems to be the biggest practical problem. All of the codes I've studied (with one exception: "Perpetual" codes) suffer from having to receive almost the full stream before you can decode it. That decoding step also involves lots of random accesses over the full stream, so you end up doing somewhere in the order of 10 random disk reads/writes per received block before you can start working on the data. That sort of latency is a death knell for many applications.

That said, I'm still optimistic that fountain codes (and their ilk) is, one day, going to be improved to the point that something based on it will become the de facto method of synchronising data sets across large groups of peer machines. It's already almost there when it comes to distributed data stores where low latency isn't so much of a requirement. It also achieves huge space savings over traditional replicated stores without sacrificing data integrity. If we could get the latency issue licked, we'd have pretty much the perfect system to build all our distributed protocols on.