Apache Oozie is a work flow engine which allows for a DAG of tasks to be run. In earlier blog entries we looked at installing/configuring Oozie, creating/executing simple work flows and finally creating a coordinator/scheduler using Oozie. Azkaban from LinkedIn is similar to Apache Oozie.
For each action (like Hive, Pig, Sqoop etc), Oozie launches an MR launcher which in turn launches the actual action. Here is a discussion from the Apache forums on the rational behind this.
In the above work flow, after the fork (in green) the Pig and Sqoop actions (in yellow) are launched by two separate MR launchers in parallel. So, at this point of time four Map slots are required (two for MR launchers, one for Pig and one for Sqoop). But, default only two Map slots are available in each node. That is, only two Map tasks will run at any point of time in a node. This is specified by the `mapred.tasktracker.map.tasks.maximum` property which defaults to 2 in the mapred-site.xml file.
The two available slots are occupied by the Oozie MR launchers and there are no more Map slots available to run the Map tasks related to Pig and Sqoop action. In this case the Job Tracker keeps on waiting for the availability of the Map slots which will never happen.
The above mentioned problem happens only in the case of Hadoop running in the pseudo distributed mode (i.e., single node/machine) with the default configurations which is usually the case of development. One solution is to increase the number of nodes and configure the Hadoop cluster in a fully distributed mode. Another solution is to bump up the `mapred.tasktracker.map.tasks.maximum` property in the mapred-site.xml as below.
In the above work flow, after the fork (in green) the Pig and Sqoop actions (in yellow) are launched by two separate MR launchers in parallel. So, at this point of time four Map slots are required (two for MR launchers, one for Pig and one for Sqoop). But, default only two Map slots are available in each node. That is, only two Map tasks will run at any point of time in a node. This is specified by the `mapred.tasktracker.map.tasks.maximum` property which defaults to 2 in the mapred-site.xml file.
The two available slots are occupied by the Oozie MR launchers and there are no more Map slots available to run the Map tasks related to Pig and Sqoop action. In this case the Job Tracker keeps on waiting for the availability of the Map slots which will never happen.
The above mentioned problem happens only in the case of Hadoop running in the pseudo distributed mode (i.e., single node/machine) with the default configurations which is usually the case of development. One solution is to increase the number of nodes and configure the Hadoop cluster in a fully distributed mode. Another solution is to bump up the `mapred.tasktracker.map.tasks.maximum` property in the mapred-site.xml as below.
<property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>4</value> </property>
No comments:
Post a Comment