World+dog agrees that Hadoop is a very fine tool with which to tackle map reduce chores, but the software has a couple of constraints, especially its reliance on the Hadoop Distributed File System (HDFS). There's nothing wrong with HDFS, but its integration with Hadoop means the software needs a dedicated cluster of computers on …
"“We abstracted out an HDFS layer but underneath that it is actually talking to lustre."
Err, Hadoop has a specific class/interface, FileSystem, designed to let anyone implement a filesystem underneath: local being a key one. All you have to do is implement it and then pass tests like FileSystemContractBaseTest to convince yourself you got it right.
While Intel make it sound like they did some heavy engineering "we abstracted out an HDFS layer", what they probably mean is they took the ASF-supported LocalFileSystem class and tweaked it to get locality information out of Lustre, then ran some (? how many?) tests to show it worked. Having them talk about the tests, that would be interesting. Ask them (or any other "we swapped HDFS for -" vendor) for that question, as only EMC/Pivotal have owned up to testing on a 1000+ node cluster
- Does Apple's iOS make you physically SICK? Try swallowing version 7.1
- Fee fie Firefox: Mozilla's lawyers probe Dell over browser install charge
- Pics Indestructible Death Stars blow up planets with glowing KILL RAY
- Video Snowden: You can't trust SPOOKS with your DATA
- Review Distro diaspora: Four flavours of Ubuntu unpacked