cascading.flow.hadoop2

Class Hadoop2MR1Planner

java.lang.Object

cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

cascading.flow.hadoop.planner.HadoopPlanner

cascading.flow.hadoop2.Hadoop2MR1Planner

public class Hadoop2MR1Planner
extends HadoopPlanner

Class Hadoop2MR1Planner is the core Hadoop MapReduce planner used by default through the Hadoop2MR1FlowConnector.

Notes:

Custom JobConf properties
A custom JobConf instance can be passed to this planner by calling HadoopPlanner.copyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf) on a map properties object before constructing a new Hadoop2MR1FlowConnector.

A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector. All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting Flow instances.

For example, properties.set("mapred.child.java.opts","-Xmx512m"); would convince Hadoop to spawn all child jvms with a heap of 512MB.

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.planner.FlowPlanner

FlowPlanner.TempTapElementFactory

Field Summary

Fields

Modifier and Type	Field and Description
static String	PLATFORM_NAME

Fields inherited from class cascading.flow.planner.FlowPlanner

checkpointTapRootPath, defaultAssertionLevel, defaultDebugLevel, defaultProperties, TRACE_PLAN_PATH, TRACE_PLAN_TRANSFORM_PATH, TRACE_STATS_PATH

Constructor Summary

Constructors

Constructor and Description
Hadoop2MR1Planner()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkPlatform(Configuration conf)
static void	copyConfiguration(Map<Object,Object> properties, Configuration configuration) Method copyJobConf adds the given JobConf values to the given properties object.
static void	copyProperties(Configuration configuration, Map<Object,Object> properties) Method copyProperties adds the given Map values to the given JobConf object.
PlannerInfo	getPlannerInfo(String registryName)

Methods inherited from class cascading.flow.hadoop.planner.HadoopPlanner

configRuleRegistryDefaults, copyJobConf, copyProperties, createFlow, createFlowStep, createJobConf, getDefaultConfig, getDefaultURIScheme, getPlatformInfo, getURIScheme, initialize, makeTempTap

Methods inherited from class cascading.flow.planner.FlowPlanner

buildFlow, createFlowElementGraph, createFlowNode, getAssertionLevel, getDebugLevel, getDefaultProperties, handleExceptionDuringPlanning, makeFlowNodeName, makeFlowStepName, makeTempTap, resolveAssemblyPlanners, resolveTails, verifyAllTaps, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifyResult, verifyResultInternal, verifySourceNotSinks, verifyTaps, verifyTraps

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLATFORM_NAME

public static final String PLATFORM_NAME

See Also:

Constant Field Values

Constructor Detail

Hadoop2MR1Planner

public Hadoop2MR1Planner()

Method Detail

copyConfiguration

public static void copyConfiguration(Map<Object,Object> properties,
                     Configuration configuration)

Method copyJobConf adds the given JobConf values to the given properties object. Use this method to pass custom default Hadoop JobConf properties to Hadoop.

Parameters:

properties - of type Map

configuration - of type JobConf

copyProperties

public static void copyProperties(Configuration configuration,
                  Map<Object,Object> properties)

Method copyProperties adds the given Map values to the given JobConf object.

Parameters:

configuration - of type JobConf

properties - of type Map

getPlannerInfo

public PlannerInfo getPlannerInfo(String registryName)

Overrides:

getPlannerInfo in class HadoopPlanner

checkPlatform

protected void checkPlatform(Configuration conf)

Overrides:

checkPlatform in class HadoopPlanner