cascading.flow.hadoop
Class MapReduceFlow
java.lang.Object
cascading.flow.BaseFlow<JobConf>
cascading.flow.hadoop.HadoopFlow
cascading.flow.hadoop.MapReduceFlow
- All Implemented Interfaces:
- Flow<JobConf>, UnitOfWork<FlowStats>
public class MapReduceFlow
- extends HadoopFlow
Class MapReduceFlow is a HadoopFlow
subclass that supports custom MapReduce jobs
pre-configured via the JobConf
object.
Use this class to allow custom MapReduce jobs to participate in the Cascade
scheduler. If
other Flow instances in the Cascade share resources with this Flow instance, all participants will be scheduled
according to their dependencies (topologically).
Set the parameter deleteSinkOnInit
to true
if the outputPath in the jobConf should be deleted before executing the MapReduce job.
MapReduceFlow assumes the underlying input and output paths are compatible with the Hfs
Tap.
If the configured JobConf instance uses some other identifier instead of Hadoop FS paths, you should override the
createSources(org.apache.hadoop.mapred.JobConf)
, createSinks(org.apache.hadoop.mapred.JobConf)
, and
createTraps(org.apache.hadoop.mapred.JobConf)
methods to properly resolve the configured paths into
usable Tap
instances. By default createTraps returns an empty collection and should probably be left alone.
Constructor Summary |
MapReduceFlow(JobConf jobConf)
Constructor MapReduceFlow creates a new MapReduceFlow instance. |
MapReduceFlow(JobConf jobConf,
boolean deleteSinkOnInit)
Constructor MapReduceFlow creates a new MapReduceFlow instance. |
MapReduceFlow(String name,
JobConf jobConf)
Constructor MapReduceFlow creates a new MapReduceFlow instance. |
MapReduceFlow(String name,
JobConf jobConf,
boolean deleteSinkOnInit)
Constructor MapReduceFlow creates a new MapReduceFlow instance. |
MapReduceFlow(String name,
JobConf jobConf,
boolean deleteSinkOnInit,
boolean stopJobsOnExit)
Constructor MapReduceFlow creates a new MapReduceFlow instance. |
Methods inherited from class cascading.flow.hadoop.HadoopFlow |
getConfig, getConfigAsProperties, getConfigCopy, getFlowProcess, getMaxNumParallelSteps, getProperty, initConfig, initFromProperties, internalClean, internalShutdown, internalStart, isPreserveTemporaryFiles, newConfig, setConfigProperty, stepsAreLocal |
Methods inherited from class cascading.flow.BaseFlow |
addListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnStarting, fireOnStopping, getCascadeID, getCascadingServices, getCheckpoints, getCheckpointsCollection, getFieldsFor, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowSteps, getFlowStepStrategy, getHolder, getID, getName, getSink, getSink, getSinkModified, getSinks, getSinksCollection, getSource, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, initialize, initializeNewJobsMap, initSteps, internalStopAllJobs, isSkipFlow, isStopJobsOnExit, logInfo, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, start, stop, toString, updateSchemes, writeDOT, writeStepsDOT |
deleteSinkOnInit
protected boolean deleteSinkOnInit
- Field deleteSinkOnInit
MapReduceFlow
@ConstructorProperties(value="jobConf")
public MapReduceFlow(JobConf jobConf)
- Constructor MapReduceFlow creates a new MapReduceFlow instance.
- Parameters:
jobConf
- of type JobConf
MapReduceFlow
@ConstructorProperties(value={"jobConf","deleteSinkOnInit"})
public MapReduceFlow(JobConf jobConf,
boolean deleteSinkOnInit)
- Constructor MapReduceFlow creates a new MapReduceFlow instance.
- Parameters:
jobConf
- of type JobConfdeleteSinkOnInit
- of type boolean
MapReduceFlow
@ConstructorProperties(value={"name","jobConf"})
public MapReduceFlow(String name,
JobConf jobConf)
- Constructor MapReduceFlow creates a new MapReduceFlow instance.
- Parameters:
name
- of type StringjobConf
- of type JobConf
MapReduceFlow
@ConstructorProperties(value={"name","jobConf","deleteSinkOnInit"})
public MapReduceFlow(String name,
JobConf jobConf,
boolean deleteSinkOnInit)
- Constructor MapReduceFlow creates a new MapReduceFlow instance.
- Parameters:
name
- of type StringjobConf
- of type JobConfdeleteSinkOnInit
- of type boolean
MapReduceFlow
@ConstructorProperties(value={"name","jobConf","deleteSinkOnInit","stopJobsOnExit"})
public MapReduceFlow(String name,
JobConf jobConf,
boolean deleteSinkOnInit,
boolean stopJobsOnExit)
- Constructor MapReduceFlow creates a new MapReduceFlow instance.
- Parameters:
name
- of type StringjobConf
- of type JobConfdeleteSinkOnInit
- of type booleanstopJobsOnExit
- of type boolean
createSources
protected Map<String,Tap> createSources(JobConf jobConf)
createSinks
protected Map<String,Tap> createSinks(JobConf jobConf)
createTraps
protected Map<String,Tap> createTraps(JobConf jobConf)
Copyright © 2007-2012 Concurrent, Inc. All Rights Reserved.