cascading.flow.hadoop
Class HadoopFlow

java.lang.Object
  extended by cascading.flow.BaseFlow<JobConf>
      extended by cascading.flow.hadoop.HadoopFlow
All Implemented Interfaces:
Flow<JobConf>, UnitOfWork<FlowStats>
Direct Known Subclasses:
MapReduceFlow, ProcessFlow

public class HadoopFlow
extends BaseFlow<JobConf>

Class HadoopFlow is the Apache Hadoop specific implementation of a Flow.

HadoopFlow must be created through a HadoopFlowConnector instance.

See Also:
HadoopFlowConnector

Nested Class Summary
 
Nested classes/interfaces inherited from class cascading.flow.BaseFlow
BaseFlow.FlowHolder
 
Field Summary
 
Fields inherited from class cascading.flow.BaseFlow
flowStats, sinks, sources, stop, stopJobsOnExit, thread
 
Fields inherited from interface cascading.flow.Flow
CASCADING_FLOW_ID
 
Constructor Summary
protected HadoopFlow()
           
  HadoopFlow(Map<Object,Object> properties, JobConf jobConf, FlowDef flowDef)
           
protected HadoopFlow(Map<Object,Object> properties, JobConf jobConf, String name)
           
 
Method Summary
 JobConf getConfig()
          Method getConfig returns the internal configuration object.
 Map<Object,Object> getConfigAsProperties()
          Method getConfiAsProperties converts the internal configuration object into a Map of key value pairs.
 JobConf getConfigCopy()
          Method getConfigCopy returns a copy of the internal configuration object.
 FlowProcess<JobConf> getFlowProcess()
           
protected  int getMaxNumParallelSteps()
           
 String getProperty(String key)
          Method getProperty returns the value associated with the given key from the underlying properties system.
protected  void initConfig(Map<Object,Object> properties, JobConf parentConfig)
          This method creates a new internal Config with the parentConfig as defaults using the properties to override the defaults.
protected  void initFromProperties(Map<Object,Object> properties)
           
protected  void internalClean(boolean force)
           
protected  void internalShutdown()
           
protected  void internalStart()
           
 boolean isPreserveTemporaryFiles()
          Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
protected  JobConf newConfig(JobConf defaultConfig)
           
protected  void setConfigProperty(JobConf config, Object key, Object value)
           
 boolean stepsAreLocal()
          Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.
 
Methods inherited from class cascading.flow.BaseFlow
addListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnStarting, fireOnStopping, getCascadeID, getCascadingServices, getCheckpoints, getCheckpointsCollection, getFieldsFor, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowSteps, getFlowStepStrategy, getHolder, getID, getName, getSink, getSink, getSinkModified, getSinks, getSinksCollection, getSource, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, initialize, initializeNewJobsMap, initSteps, internalStopAllJobs, isSkipFlow, isStopJobsOnExit, logInfo, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, start, stop, toString, updateSchemes, writeDOT, writeStepsDOT
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

HadoopFlow

protected HadoopFlow()

HadoopFlow

protected HadoopFlow(Map<Object,Object> properties,
                     JobConf jobConf,
                     String name)

HadoopFlow

public HadoopFlow(Map<Object,Object> properties,
                  JobConf jobConf,
                  FlowDef flowDef)
Method Detail

initFromProperties

protected void initFromProperties(Map<Object,Object> properties)
Overrides:
initFromProperties in class BaseFlow<JobConf>

initConfig

protected void initConfig(Map<Object,Object> properties,
                          JobConf parentConfig)
Description copied from class: BaseFlow
This method creates a new internal Config with the parentConfig as defaults using the properties to override the defaults.

Specified by:
initConfig in class BaseFlow<JobConf>

setConfigProperty

protected void setConfigProperty(JobConf config,
                                 Object key,
                                 Object value)
Specified by:
setConfigProperty in class BaseFlow<JobConf>

newConfig

protected JobConf newConfig(JobConf defaultConfig)
Specified by:
newConfig in class BaseFlow<JobConf>

getConfig

public JobConf getConfig()
Description copied from interface: Flow
Method getConfig returns the internal configuration object.

Any changes to this object will not be reflected in child steps. See FlowConnector for setting default properties visible to children. Or see FlowStepStrategy for setting properties on individual steps before they are executed.

Returns:
the default configuration of this Flow

getConfigCopy

public JobConf getConfigCopy()
Description copied from interface: Flow
Method getConfigCopy returns a copy of the internal configuration object. This object can be safely modified.

Returns:
a copy of the default configuration of this Flow

getConfigAsProperties

public Map<Object,Object> getConfigAsProperties()
Description copied from interface: Flow
Method getConfiAsProperties converts the internal configuration object into a Map of key value pairs.

Returns:
a Map of key/value pairs

getProperty

public String getProperty(String key)
Method getProperty returns the value associated with the given key from the underlying properties system.

Parameters:
key - of type String
Returns:
String

getFlowProcess

public FlowProcess<JobConf> getFlowProcess()

isPreserveTemporaryFiles

public boolean isPreserveTemporaryFiles()
Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.

Returns:
the preserveTemporaryFiles (type boolean) of this Flow object.

internalStart

protected void internalStart()
Specified by:
internalStart in class BaseFlow<JobConf>

stepsAreLocal

public boolean stepsAreLocal()
Description copied from interface: Flow
Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.

Returns:
boolean

internalClean

protected void internalClean(boolean force)
Specified by:
internalClean in class BaseFlow<JobConf>

internalShutdown

protected void internalShutdown()
Specified by:
internalShutdown in class BaseFlow<JobConf>

getMaxNumParallelSteps

protected int getMaxNumParallelSteps()
Specified by:
getMaxNumParallelSteps in class BaseFlow<JobConf>


Copyright © 2007-2012 Concurrent, Inc. All Rights Reserved.