MRv2Resources

'Next Generation MR' or 'NextGen MR' or 'MRv2' or 'MR2' is a major revamp of the MapReduce engine and will part of the 0.23 release. MRv1 or the old MapReduce engine will be not be supported in 0.23 release. The underlying engine has been revamped in 0.23, but the API to interface with the engine remains the same. So, the existing MapReduce code for MRv1 engine should run without modifications on MRv2.

The architecture, information for building and running MRv2 is spread across and this blog entry will try to consolidate and present all the information available on MRv2. I will keep-on updating this blog entry as I get more information about MRv2, instead of creating a new one. So, bookmark this and check it often :).

MRv1 vs MRv2

http://www.ibm.com/developerworks/library/bd-hadoopyarn/index.html

Current Status

http://hortonworks.com/blog/apache-hadoop-yarn-background-and-an-overview/  - 9th August, 2012

http://www.hortonworks.com/update-on-apache-hadoop-0-23/ - 27th September, 2011

http://www.cloudera.com/blog/2011/11/apache-hadoop-0-23-0-has-been-released/ - 15th November, 2011

http://hortonworks.com/apache-hadoop-is-here/ - 16th November, 2011

Home Page

http://hadoop.apache.org/common/docs/r0.23.0/

Architecture

MRv2 Introduction

The Hadoop Map-Reduce Capacity Scheduler

The Next Generation of Apache Hadoop MapReduce

Next Generation of Apache Hadoop MapReduce – The Scheduler

Detailed document on MRv2

HortonWorks Series

http://hortonworks.com/blog/apache-hadoop-yarn-background-and-an-overview/

http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/

http://hortonworks.com/blog/apache-hadoop-yarn-resourcemanager/

http://hortonworks.com/blog/apache-hadoop-yarn-nodemanager/

Developing applications on YARN

http://continuuity.github.io/weave/

Applications ported to Yarn

A lot of applications are being migrated to YARN.

http://wiki.apache.org/hadoop/PoweredByYarn

Presentation

Quick view of MRv2

JIRAs

https://issues.apache.org/jira/browse/MAPREDUCE-279

Books

3rd edition of the Hadoop : The Definitive Guide due on April 30th, 2012 has a chapter on MRv2.

Videos

Next Generation Hadoop MapReduce by Arun C. Murthy

Code

http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23/

Building from code and running a sample

On a single node - http://www.thecloudavenue.com/2012/01/getting-started-with-nextgen-mapreduce.html

On a cluster - http://www.thecloudavenue.com/2012/01/getting-started-with-nextgen-mapreduce_11.html

http://www.cloudera.com/blog/2011/11/building-and-deploying-mr2/

http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/