GoMR: A MapReduce Framework for Go
Date:
More information here and here.
Slides
Code
Abstract
Running Hadoop or Spark locally can be a nightmare. There are many parameters to tweak, and Java is slow, memory inefficient and the JVM imposes heap caps.
In this talk, I present an efficient MR framework, GoMR, that beats Spark on the canonical wordcount app by an order of magnitude.
Description
Frustrated trying to run/debug MapReduce applications on my laptop, I decided to spin my own MR framework to solve the problems of efficiency, configuration, and error messages. Go is a compiled language and has inherent support for distribution in the form of channels. I took advantage of these two strengths to create GoMR.
In this talk, I give an overview of the framework, the programming style I used to create it, and an evaluation of the framework against a current state of the art system, Apache Spark.
A more complete description can be found here, and its evaluation here.
I hope this work will encourage a move away from the cumbersome JVM, and spark a next-generation of efficient distributed compute frameworks.