public class HadoopPlanner extends FlowPlanner<HadoopFlow,JobConf>
FlowConnector
sub-class.
Notes:
Custom JobConf propertiescopyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf)
on a map properties object before constructing a new FlowConnector
sub-class.
A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector.
All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting
Flow instances.
For example, properties.set("mapred.child.java.opts","-Xmx512m");
would convince Hadoop
to spawn all child jvms with a heap of 512MB.FlowPlanner.TempTapElementFactory
Modifier and Type | Field and Description |
---|---|
static String |
PLATFORM_NAME |
checkpointTapRootPath, defaultAssertionLevel, defaultDebugLevel, defaultProperties, TRACE_PLAN_PATH, TRACE_PLAN_TRANSFORM_PATH, TRACE_STATS_PATH
Constructor and Description |
---|
HadoopPlanner() |
Modifier and Type | Method and Description |
---|---|
protected void |
checkPlatform(Configuration conf) |
void |
configRuleRegistryDefaults(RuleRegistry ruleRegistry) |
static void |
copyJobConf(Map<Object,Object> properties,
JobConf jobConf)
Method copyJobConf adds the given JobConf values to the given properties object.
|
static void |
copyProperties(JobConf jobConf,
Map<Object,Object> properties)
Method copyProperties adds the given Map values to the given JobConf object.
|
protected HadoopFlow |
createFlow(FlowDef flowDef) |
static JobConf |
createJobConf(Map<Object,Object> properties)
Method createJobConf returns a new JobConf instance using the values in the given properties argument.
|
JobConf |
getDefaultConfig() |
URI |
getDefaultURIScheme(Tap tap) |
FlowStepFactory<JobConf> |
getFlowStepFactory() |
PlannerInfo |
getPlannerInfo(String registryName) |
PlatformInfo |
getPlatformInfo() |
URI |
getURIScheme(Tap tap) |
void |
initialize(FlowConnector flowConnector,
Map<Object,Object> properties) |
protected Tap |
makeTempTap(String prefix,
String name) |
buildFlow, createFlowElementGraph, getAssertionLevel, getDebugLevel, getDefaultProperties, getFlowNodeFactory, handleExceptionDuringPlanning, makeTempTap, resolveAssemblyPlanners, resolveTails, verifyAllTaps, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifyResult, verifyResultInternal, verifySourceNotSinks, verifyTaps, verifyTraps
public static final String PLATFORM_NAME
public HadoopPlanner()
public static void copyJobConf(Map<Object,Object> properties, JobConf jobConf)
properties
- of type MapjobConf
- of type JobConfpublic static JobConf createJobConf(Map<Object,Object> properties)
properties
- of type Mappublic static void copyProperties(JobConf jobConf, Map<Object,Object> properties)
jobConf
- of type JobConfproperties
- of type Mappublic PlannerInfo getPlannerInfo(String registryName)
getPlannerInfo
in class FlowPlanner<HadoopFlow,JobConf>
public JobConf getDefaultConfig()
getDefaultConfig
in class FlowPlanner<HadoopFlow,JobConf>
public PlatformInfo getPlatformInfo()
getPlatformInfo
in class FlowPlanner<HadoopFlow,JobConf>
public void initialize(FlowConnector flowConnector, Map<Object,Object> properties)
initialize
in class FlowPlanner<HadoopFlow,JobConf>
public void configRuleRegistryDefaults(RuleRegistry ruleRegistry)
configRuleRegistryDefaults
in class FlowPlanner<HadoopFlow,JobConf>
protected void checkPlatform(Configuration conf)
protected HadoopFlow createFlow(FlowDef flowDef)
createFlow
in class FlowPlanner<HadoopFlow,JobConf>
public FlowStepFactory<JobConf> getFlowStepFactory()
getFlowStepFactory
in class FlowPlanner<HadoopFlow,JobConf>
public URI getDefaultURIScheme(Tap tap)
public URI getURIScheme(Tap tap)
protected Tap makeTempTap(String prefix, String name)
makeTempTap
in class FlowPlanner<HadoopFlow,JobConf>
Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.