cascading.flow
Class FlowStep

java.lang.Object
  cascading.flow.FlowStep

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

MapReduceFlowStep

public class FlowStep
extends Object
implements Serializable

Class FlowStep is an internal representation of a given Job to be executed on a remote cluster. During planning, pipe assemblies are broken down into "steps" and encapsulated in this class.

FlowSteps are submitted in order of dependency. If two or more steps do not share the same dependencies and all can be scheduled simultaneously, the getSubmitPriority() value determines the order in which all steps will be submitted for execution. The default submit priority is 5.

This class is for internal use, there are no stable public methods.

See Also:

Serialized Form

Field Summary

protected Tap

sink Field sink

Constructor Summary

protected

FlowStep(String name, int id)

Method Summary

void	clean(JobConf jobConf) Method clean removes any temporary files used by this FlowStep instance.
boolean	containsPipeNamed(String pipeName)
protected FlowStepJob	createFlowStepJob(JobConf parentConf)
boolean	equals(Object object)
Collection<Operation>	getAllOperations()
Group	getGroup()
int	getID() Method getId returns the id of this FlowStep object.
protected JobConf	getJobConf()
protected JobConf	getJobConf(JobConf parentConf)
Tap	getMapperTrap(String name)
Map<String,Tap>	getMapperTraps()
String	getName() Method getName returns the name of this FlowStep object.
FlowElement	getNextFlowElement(Scope scope)
Scope	getNextScope(FlowElement flowElement) Method getNextScope returns the next Scope instance in the graph.
Set<Scope>	getNextScopes(FlowElement flowElement)
String	getParentFlowName() Method getParentFlowName returns the parentFlowName of this FlowStep object.
Set<Scope>	getPreviousScopes(FlowElement flowElement) Method getPreviousScopes returns the previous Scope instances.
Map<Object,Object>	getProperties() Method getProperties returns the properties of this FlowStep object.
Tap	getReducerTrap(String name)
Map<String,Tap>	getReducerTraps()
String	getSourceName(Tap source)
String	getStepName() Method getStepName returns the stepName of this FlowStep object.
int	getSubmitPriority() Method getSubmitPriority returns the submitPriority of this FlowStep object.
int	hashCode()
boolean	hasProperties() Method hasProperties returns true if there are properties associated with this FlowStep.
protected boolean	isDebugEnabled()
protected boolean	isInfoEnabled()
protected void	logDebug(String message)
protected void	logError(String message, Throwable throwable)
protected void	logInfo(String message)
protected void	logWarn(String message)
protected void	logWarn(String message, Throwable throwable)
TupleEntryIterator	openSinkForRead(JobConf conf)
TapIterator	openSourceForRead(JobConf conf)
protected void	setGroup(Group group)
void	setName(String name)
void	setParentFlowName(String parentFlowName) Method setParentFlowName sets the parentFlowName of this FlowStep object.
void	setProperties(Map<Object,Object> properties) Method setProperties sets the properties of this FlowStep object.
void	setSubmitPriority(int submitPriority) Method setSubmitPriority sets the submitPriority of this FlowStep object.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

sink

protected Tap sink

Field sink

Constructor Detail

FlowStep

protected FlowStep(String name,
                   int id)

Method Detail

getID

public int getID()

Method getId returns the id of this FlowStep object.

Returns:

the id (type int) of this FlowStep object.

getName

public String getName()

Method getName returns the name of this FlowStep object.

Returns:

the name (type String) of this FlowStep object.

setName

public void setName(String name)

getParentFlowName

public String getParentFlowName()

Method getParentFlowName returns the parentFlowName of this FlowStep object.

Returns:

the parentFlowName (type Flow) of this FlowStep object.

setParentFlowName

public void setParentFlowName(String parentFlowName)

Method setParentFlowName sets the parentFlowName of this FlowStep object.

Parameters:

parentFlowName - the parentFlowName of this FlowStep object.

getStepName

public String getStepName()

Method getStepName returns the stepName of this FlowStep object.

Returns:

the stepName (type String) of this FlowStep object.

getSubmitPriority

public int getSubmitPriority()

Method getSubmitPriority returns the submitPriority of this FlowStep object.

10 is lowest, 1 is the highest, 5 is the default.

Returns:

the submitPriority (type int) of this FlowStep object.

setSubmitPriority

public void setSubmitPriority(int submitPriority)

Method setSubmitPriority sets the submitPriority of this FlowStep object.

10 is lowest, 1 is the highest, 5 is the default.

Parameters:

submitPriority - the submitPriority of this FlowStep object.

getGroup

public Group getGroup()

setGroup

protected void setGroup(Group group)

getMapperTraps

public Map<String,Tap> getMapperTraps()

getReducerTraps

public Map<String,Tap> getReducerTraps()

getProperties

public Map<Object,Object> getProperties()

Method getProperties returns the properties of this FlowStep object.

Returns:

the properties (type Map) of this FlowStep object.

setProperties

public void setProperties(Map<Object,Object> properties)

Method setProperties sets the properties of this FlowStep object.

Parameters:

properties - the properties of this FlowStep object.

hasProperties

public boolean hasProperties()

Method hasProperties returns true if there are properties associated with this FlowStep.

Returns:

boolean

getJobConf

protected JobConf getJobConf()
                      throws IOException

Throws:

IOException

getJobConf

protected JobConf getJobConf(JobConf parentConf)
                      throws IOException

Throws:

IOException

openSourceForRead

public TapIterator openSourceForRead(JobConf conf)
                              throws IOException

Throws:

IOException

openSinkForRead

public TupleEntryIterator openSinkForRead(JobConf conf)
                                   throws IOException

Throws:

IOException

getMapperTrap

public Tap getMapperTrap(String name)

getReducerTrap

public Tap getReducerTrap(String name)

getPreviousScopes

public Set<Scope> getPreviousScopes(FlowElement flowElement)

Method getPreviousScopes returns the previous Scope instances. If the flowElement is a Group (specifically a CoGroup), there will be more than one instance.

Parameters:

flowElement - of type FlowElement

Returns:

Set

getNextScope

public Scope getNextScope(FlowElement flowElement)

Method getNextScope returns the next Scope instance in the graph. There will always only be one next.

Parameters:

flowElement - of type FlowElement

Returns:

Scope

getNextScopes

public Set<Scope> getNextScopes(FlowElement flowElement)

getNextFlowElement

public FlowElement getNextFlowElement(Scope scope)

getSourceName

public String getSourceName(Tap source)

getAllOperations

public Collection<Operation> getAllOperations()

containsPipeNamed

public boolean containsPipeNamed(String pipeName)

clean

public void clean(JobConf jobConf)

Method clean removes any temporary files used by this FlowStep instance. It will log any IOExceptions thrown.

Parameters:

jobConf - of type JobConf

equals

public boolean equals(Object object)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object

createFlowStepJob

protected FlowStepJob createFlowStepJob(JobConf parentConf)
                                 throws IOException

Throws:

IOException

isInfoEnabled

protected final boolean isInfoEnabled()

isDebugEnabled

protected final boolean isDebugEnabled()

logDebug

protected void logDebug(String message)

logInfo

protected void logInfo(String message)

logWarn

protected void logWarn(String message)

logWarn

protected void logWarn(String message,
                       Throwable throwable)

logError

protected void logError(String message,
                        Throwable throwable)