cascading.flow.planner
Class BaseFlowStep<Config>

java.lang.Object
  cascading.flow.planner.BaseFlowStep<Config>

All Implemented Interfaces:

FlowStep<Config>, Serializable

public abstract class BaseFlowStep<Config>
extends Object
implements Serializable, FlowStep<Config>

Class FlowStep is an internal representation of a given Job to be executed on a remote cluster. During planning, pipe assemblies are broken down into "steps" and encapsulated in this class.

FlowSteps are submitted in order of dependency. If two or more steps do not share the same dependencies and all can be scheduled simultaneously, the getSubmitPriority() value determines the order in which all steps will be submitted for execution. The default submit priority is 5.

This class is for internal use, there are no stable public methods.

See Also:

Serialized Form

Field Summary

protected Map<HashJoin,Set<Tap>>	accumulatedSourcesByJoin
protected Map<Tap,Set<String>>	sinks Field sink
protected Map<Tap,Set<String>>	sources Field sources
protected Map<HashJoin,Tap>	streamedSourceByJoin
protected Tap	tempSink Field tempSink

Fields inherited from interface cascading.flow.FlowStep

CASCADING_FLOW_STEP_ID

Constructor Summary

protected

BaseFlowStep(String name, int stepNum)

Method Summary

void	addAccumulatedSourceFor(HashJoin join, Tap accumulatedSource)
void	addGroup(Group group)
void	addListener(FlowStepListener flowStepListener) Method addListener registers the given FlowStepListener with this instance.
void	addSink(String name, Tap sink)
void	addSource(String name, Tap source)
void	addStreamedSourceFor(HashJoin join, Tap streamedSource)
void	clean()
abstract void	clean(Config config)
protected Throwable	commitSinks()
boolean	containsPipeNamed(String pipeName) Returns true if this FlowStep contains a pipe/branch with the given name.
protected ClientState	createClientState(FlowProcess flowProcess)
protected abstract FlowStepJob	createFlowStepJob(FlowProcess<Config> flowProcess, Config parentConfig)
boolean	equals(Object object)
protected void	fireOnCompleted()
protected void	fireOnRunning()
protected void	fireOnStarting()
protected void	fireOnStopping()
protected void	fireOnThrowable(Throwable throwable)
Set<Tap>	getAllAccumulatedSources()
Collection<Operation>	getAllOperations()
Config	getConfig() Method getConfig returns the current initialized configuration.
Flow<Config>	getFlow()
String	getFlowID()
String	getFlowName() Method getParentFlowName returns the parentFlowName of this FlowStep object.
FlowStepJob<Config>	getFlowStepJob(FlowProcess<Config> flowProcess, Config parentConfig)
FlowStepStats	getFlowStepStats()
org.jgrapht.graph.SimpleDirectedGraph<FlowElement,Scope>	getGraph()
Group	getGroup()
List<Group>	getGroups()
String	getID() Method getId returns the id of this FlowStep object.
protected abstract Config	getInitializedConfig(FlowProcess<Config> flowProcess, Config parentConfig)
Set<Tap>	getJoinTributariesBetween(FlowElement from, FlowElement to)
String	getName() Method getName returns the name of this FlowStep object.
FlowElement	getNextFlowElement(Scope scope)
Scope	getNextScope(FlowElement flowElement) Method getNextScope returns the next Scope instance in the graph.
Set<Scope>	getNextScopes(FlowElement flowElement)
Set<Scope>	getPreviousScopes(FlowElement flowElement) Method getPreviousScopes returns the previous Scope instances.
Scope	getScopeFor(FlowElement sourceElement, FlowElement targetElement)
Tap	getSink()
Set<String>	getSinkName(Tap sink)
Set<Tap>	getSinks()
Tap	getSinkWith(String identifier)
Set<String>	getSourceName(Tap source)
Set<Tap>	getSources()
Tap	getSourceWith(String identifier)
String	getStepDisplayName() Method getStepDisplayName returns the stepDisplayName of this FlowStep object.
protected String	getStepDisplayName(int idLength)
int	getStepNum()
Map<HashJoin,Tap>	getStreamedSourceByJoin()
int	getSubmitPriority() Method getSubmitPriority returns the submitPriority of this FlowStep object.
List<FlowElement>	getSuccessors(FlowElement element)
org.jgrapht.traverse.TopologicalOrderIterator<FlowElement,Scope>	getTopologicalOrderIterator()
int	hashCode()
boolean	hasListeners() Method hasListeners returns true if FlowStepListener instances have been registered.
protected void	initConfFromProcessConfigDef(ConfigDef.Setter setter)
boolean	isDebugEnabled()
boolean	isInfoEnabled()
void	logDebug(String message)
void	logError(String message, Throwable throwable)
void	logInfo(String message)
void	logWarn(String message)
void	logWarn(String message, Throwable throwable)
protected Throwable	prepareResources()
boolean	removeListener(FlowStepListener flowStepListener) Method removeListener removes the given flowStepListener from this instance.
protected Throwable	rollbackSinks()
protected void	setConf(Config conf)
void	setFlow(Flow<Config> flow)
protected void	setFlowName(String flowName)
void	setSubmitPriority(int submitPriority) Method setSubmitPriority sets the submitPriority of this FlowStep object.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface cascading.flow.FlowStep

getTrap, getTraps

Field Detail

sources

protected final Map<Tap,Set<String>> sources

Field sources

sinks

protected final Map<Tap,Set<String>> sinks

Field sink

tempSink

protected Tap tempSink

Field tempSink

streamedSourceByJoin

protected final Map<HashJoin,Tap> streamedSourceByJoin

accumulatedSourcesByJoin

protected final Map<HashJoin,Set<Tap>> accumulatedSourcesByJoin

Constructor Detail

BaseFlowStep

protected BaseFlowStep(String name,
                       int stepNum)

Method Detail

getID

public String getID()

Description copied from interface: FlowStep

Method getId returns the id of this FlowStep object.

Specified by:

getID in interface FlowStep<Config>

Returns:

the id (type int) of this FlowStep object.

getStepNum

public int getStepNum()

Specified by:

getStepNum in interface FlowStep<Config>

getName

public String getName()

Description copied from interface: FlowStep

Method getName returns the name of this FlowStep object.

Specified by:

getName in interface FlowStep<Config>

Returns:

the name (type String) of this FlowStep object.

setFlow

public void setFlow(Flow<Config> flow)

getFlow

public Flow<Config> getFlow()

Specified by:

getFlow in interface FlowStep<Config>

getFlowID

public String getFlowID()

Specified by:

getFlowID in interface FlowStep<Config>

getFlowName

public String getFlowName()

Description copied from interface: FlowStep

Method getParentFlowName returns the parentFlowName of this FlowStep object.

Specified by:

getFlowName in interface FlowStep<Config>

Returns:

the parentFlowName (type Flow) of this FlowStep object.

setFlowName

protected void setFlowName(String flowName)

getConfig

public Config getConfig()

Description copied from interface: FlowStep

Method getConfig returns the current initialized configuration.

The returned configuration is mutable and may be changed prior to this step being started or submitted.

Specified by:

getConfig in interface FlowStep<Config>

Returns:

the current initialized configuration

setConf

protected void setConf(Config conf)

getStepDisplayName

public String getStepDisplayName()

Description copied from interface: FlowStep

Method getStepDisplayName returns the stepDisplayName of this FlowStep object.

Specified by:

getStepDisplayName in interface FlowStep<Config>

Returns:

the stepName (type String) of this FlowStep object.

getStepDisplayName

protected String getStepDisplayName(int idLength)

getSubmitPriority

public int getSubmitPriority()

Description copied from interface: FlowStep

Method getSubmitPriority returns the submitPriority of this FlowStep object.

10 is lowest, 1 is the highest, 5 is the default.

Specified by:

getSubmitPriority in interface FlowStep<Config>

Returns:

the submitPriority (type int) of this FlowStep object.

setSubmitPriority

public void setSubmitPriority(int submitPriority)

Description copied from interface: FlowStep

Method setSubmitPriority sets the submitPriority of this FlowStep object.

10 is lowest, 1 is the highest, 5 is the default.

Specified by:

setSubmitPriority in interface FlowStep<Config>

Parameters:

submitPriority - the submitPriority of this FlowStep object.

getFlowStepStats

public FlowStepStats getFlowStepStats()

Specified by:

getFlowStepStats in interface FlowStep<Config>

getGraph

public org.jgrapht.graph.SimpleDirectedGraph<FlowElement,Scope> getGraph()

getGroup

public Group getGroup()

Specified by:

getGroup in interface FlowStep<Config>

getGroups

public List<Group> getGroups()

Specified by:

getGroups in interface FlowStep<Config>

addGroup

public void addGroup(Group group)

getStreamedSourceByJoin

public Map<HashJoin,Tap> getStreamedSourceByJoin()

Specified by:

getStreamedSourceByJoin in interface FlowStep<Config>

addStreamedSourceFor

public void addStreamedSourceFor(HashJoin join,
                                 Tap streamedSource)

getAllAccumulatedSources

public Set<Tap> getAllAccumulatedSources()

Specified by:

getAllAccumulatedSources in interface FlowStep<Config>

addAccumulatedSourceFor

public void addAccumulatedSourceFor(HashJoin join,
                                    Tap accumulatedSource)

addSource

public void addSource(String name,
                      Tap source)

addSink

public void addSink(String name,
                    Tap sink)

getSources

public Set<Tap> getSources()

Specified by:

getSources in interface FlowStep<Config>

getSinks

public Set<Tap> getSinks()

Specified by:

getSinks in interface FlowStep<Config>

getSink

public Tap getSink()

Specified by:

getSink in interface FlowStep<Config>

getSourceName

public Set<String> getSourceName(Tap source)

Specified by:

getSourceName in interface FlowStep<Config>

getSinkName

public Set<String> getSinkName(Tap sink)

Specified by:

getSinkName in interface FlowStep<Config>

getSourceWith

public Tap getSourceWith(String identifier)

Specified by:

getSourceWith in interface FlowStep<Config>

getSinkWith

public Tap getSinkWith(String identifier)

Specified by:

getSinkWith in interface FlowStep<Config>

prepareResources

protected Throwable prepareResources()

commitSinks

protected Throwable commitSinks()

rollbackSinks

protected Throwable rollbackSinks()

getInitializedConfig

protected abstract Config getInitializedConfig(FlowProcess<Config> flowProcess,
                                               Config parentConfig)

getPreviousScopes

public Set<Scope> getPreviousScopes(FlowElement flowElement)

Method getPreviousScopes returns the previous Scope instances. If the flowElement is a Group (specifically a CoGroup), there will be more than one instance.

Parameters:

flowElement - of type FlowElement

Returns:

Set

getNextScope

public Scope getNextScope(FlowElement flowElement)

Method getNextScope returns the next Scope instance in the graph. There will always only be one next.

Parameters:

flowElement - of type FlowElement

Returns:

Scope

getScopeFor

public Scope getScopeFor(FlowElement sourceElement,
                         FlowElement targetElement)

getNextScopes

public Set<Scope> getNextScopes(FlowElement flowElement)

getNextFlowElement

public FlowElement getNextFlowElement(Scope scope)

getTopologicalOrderIterator

public org.jgrapht.traverse.TopologicalOrderIterator<FlowElement,Scope> getTopologicalOrderIterator()

getSuccessors

public List<FlowElement> getSuccessors(FlowElement element)

getJoinTributariesBetween

public Set<Tap> getJoinTributariesBetween(FlowElement from,
                                          FlowElement to)

getAllOperations

public Collection<Operation> getAllOperations()

containsPipeNamed

public boolean containsPipeNamed(String pipeName)

Description copied from interface: FlowStep

Returns true if this FlowStep contains a pipe/branch with the given name.

Specified by:

containsPipeNamed in interface FlowStep<Config>

Parameters:

pipeName - the name of the Pipe

Returns:

a boolean

clean

public void clean()

clean

public abstract void clean(Config config)

hasListeners

public boolean hasListeners()

Description copied from interface: FlowStep

Method hasListeners returns true if FlowStepListener instances have been registered.

Specified by:

hasListeners in interface FlowStep<Config>

Returns:

boolean

addListener

public void addListener(FlowStepListener flowStepListener)

Description copied from interface: FlowStep

Method addListener registers the given FlowStepListener with this instance.

Specified by:

addListener in interface FlowStep<Config>

Parameters:

flowStepListener - of type flowStepListener

removeListener

public boolean removeListener(FlowStepListener flowStepListener)

Description copied from interface: FlowStep

Method removeListener removes the given flowStepListener from this instance.

Specified by:

removeListener in interface FlowStep<Config>

Parameters:

flowStepListener - of type FlowStepListener

Returns:

true if the listener was removed

fireOnCompleted

protected void fireOnCompleted()

fireOnThrowable

protected void fireOnThrowable(Throwable throwable)

fireOnStopping

protected void fireOnStopping()

fireOnStarting

protected void fireOnStarting()

fireOnRunning

protected void fireOnRunning()

equals

public boolean equals(Object object)

Overrides:

equals in class Object

createClientState

protected ClientState createClientState(FlowProcess flowProcess)

getFlowStepJob

public FlowStepJob<Config> getFlowStepJob(FlowProcess<Config> flowProcess,
                                          Config parentConfig)

createFlowStepJob

protected abstract FlowStepJob createFlowStepJob(FlowProcess<Config> flowProcess,
                                                 Config parentConfig)

initConfFromProcessConfigDef

protected void initConfFromProcessConfigDef(ConfigDef.Setter setter)

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object

isInfoEnabled

public final boolean isInfoEnabled()

isDebugEnabled

public final boolean isDebugEnabled()

logDebug

public void logDebug(String message)

logInfo

public void logInfo(String message)

logWarn

public void logWarn(String message)

logWarn

public void logWarn(String message,
                    Throwable throwable)

logError

public void logError(String message,
                     Throwable throwable)