cascading.flow.hadoop

Class HadoopFlow

java.lang.Object

cascading.flow.BaseFlow<JobConf>

cascading.flow.hadoop.HadoopFlow

All Implemented Interfaces:

Flow<JobConf>, UnitOfWork<FlowStats>, ProcessLogger

Direct Known Subclasses:

MapReduceFlow

public class HadoopFlow
extends BaseFlow<JobConf>

Class HadoopFlow is the Apache Hadoop specific implementation of a Flow.

HadoopFlow must be created through a FlowConnector sub-class instance.

If classpath paths are provided on the FlowDef, the Hadoop distributed cache mechanism will be used to augment the remote classpath.

Any path elements that are relative will be uploaded to HDFS, and the HDFS URI will be used on the JobConf. Note all paths are added as "files" to the JobConf, not archives, so they aren't needlessly uncompressed cluster side.

See Also:

FlowConnector

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.BaseFlow

BaseFlow.FlowHolder

Field Summary

Fields inherited from class cascading.flow.BaseFlow

flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread

Fields inherited from interface cascading.flow.Flow

CASCADING_FLOW_ID

Fields inherited from interface cascading.util.ProcessLogger

NULL

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HadoopFlow()
	HadoopFlow(PlatformInfo platformInfo, Map<Object,Object> properties, JobConf jobConf, FlowDef flowDef)
protected	HadoopFlow(PlatformInfo platformInfo, Map<Object,Object> properties, JobConf jobConf, String name, Map<String,String> flowDescriptor)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	copyToDistributedCache()
JobConf	getConfig()
Map<Object,Object>	getConfigAsProperties()
JobConf	getConfigCopy()
FlowProcess<JobConf>	getFlowProcess()
protected int	getMaxNumParallelSteps()
String	getProperty(String key) Method getProperty returns the value associated with the given key from the underlying properties system.
protected long	getTotalSliceCPUMilliSeconds()
protected void	initConfig(Map<Object,Object> properties, JobConf parentConfig)
protected void	initFromProperties(Map<Object,Object> properties)
protected void	internalClean(boolean stop)
protected void	internalShutdown()
protected void	internalStart()
boolean	isPreserveTemporaryFiles() Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
protected JobConf	newConfig(JobConf defaultConfig)
protected void	registerHadoopShutdownHook()
protected void	setConfigProperty(JobConf config, Object key, Object value)
boolean	stepsAreLocal()

Methods inherited from class cascading.flow.BaseFlow

addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowSteps, getFlowStepStrategy, getHolder, getID, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, start, stop, toString, updateSchemes, writeDOT, writeStepsDOT

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

HadoopFlow

protected HadoopFlow()

HadoopFlow

protected HadoopFlow(PlatformInfo platformInfo,
          Map<Object,Object> properties,
          JobConf jobConf,
          String name,
          Map<String,String> flowDescriptor)

HadoopFlow

public HadoopFlow(PlatformInfo platformInfo,
          Map<Object,Object> properties,
          JobConf jobConf,
          FlowDef flowDef)

Method Detail

initFromProperties

protected void initFromProperties(Map<Object,Object> properties)

Overrides:

initFromProperties in class BaseFlow<JobConf>

initConfig

protected void initConfig(Map<Object,Object> properties,
              JobConf parentConfig)

Specified by:

initConfig in class BaseFlow<JobConf>

setConfigProperty

protected void setConfigProperty(JobConf config,
                     Object key,
                     Object value)

Specified by:

setConfigProperty in class BaseFlow<JobConf>

newConfig

protected JobConf newConfig(JobConf defaultConfig)

Specified by:

newConfig in class BaseFlow<JobConf>

getConfig

@ProcessConfiguration
public JobConf getConfig()

getConfigCopy

public JobConf getConfigCopy()

getConfigAsProperties

public Map<Object,Object> getConfigAsProperties()

getProperty

public String getProperty(String key)

Method getProperty returns the value associated with the given key from the underlying properties system.

Parameters:

key - of type String

Returns:

String

getFlowProcess

public FlowProcess<JobConf> getFlowProcess()

isPreserveTemporaryFiles

public boolean isPreserveTemporaryFiles()

Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.

Returns:

the preserveTemporaryFiles (type boolean) of this Flow object.

internalStart

protected void internalStart()

Specified by:

internalStart in class BaseFlow<JobConf>

registerHadoopShutdownHook

protected void registerHadoopShutdownHook()

copyToDistributedCache

protected void copyToDistributedCache()

stepsAreLocal

public boolean stepsAreLocal()

internalClean

protected void internalClean(boolean stop)

Specified by:

internalClean in class BaseFlow<JobConf>

internalShutdown

protected void internalShutdown()

Specified by:

internalShutdown in class BaseFlow<JobConf>

getMaxNumParallelSteps

protected int getMaxNumParallelSteps()

Specified by:

getMaxNumParallelSteps in class BaseFlow<JobConf>

getTotalSliceCPUMilliSeconds

protected long getTotalSliceCPUMilliSeconds()

Overrides:

getTotalSliceCPUMilliSeconds in class BaseFlow<JobConf>