cascading.flow

Class FlowProcess<Config>

java.lang.Object

cascading.flow.FlowProcess<Config>

Direct Known Subclasses:

FlowProcess.NullFlowProcess, FlowProcessWrapper

public abstract class FlowProcess<Config>
extends Object

FlowProcess implementations provide a call-back interface into the current computing system. Each Operation is given a reference to a particular implementation, allowing it to get configuration properties, send a "keep alive" ping, or to set a counter value.

Depending on the underlying system, FlowProcess instances are not continuous across all operations in a Flow. Thus, a call to increment(Enum, long) may start incrementing from zero if the operation making the call belongs to a subsequent 'job' or 'step' from any previous operations calling increment.

A FlowProcess is roughly a child of FlowSession. FlowSession is roughly one to one with a particular Flow. And every FlowSession will have one or more FlowProcesses.

See Also:

FlowSession

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FlowProcess.NullFlowProcess

Field Summary

Fields

Modifier and Type	Field and Description
static FlowProcess	NULL Field NULL is a noop implementation of FlowSession.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	FlowProcess()
protected	FlowProcess(FlowProcess<Config> flowProcess) Copy constructor.
protected	FlowProcess(FlowSession currentSession)

Method Summary

Methods

Modifier and Type	Method and Description
void	closeTrapCollectors()
abstract <C> C	copyConfig(C config)
abstract FlowProcess<Config>	copyWith(Config config)
abstract <C> Map<String,String>	diffConfigIntoMap(C defaultConfig, C updatedConfig)
Boolean	getBooleanProperty(String key) Method getBooleanProperty should be used to return configuration parameters from the underlying system.
boolean	getBooleanProperty(String key, boolean defaultValue) Method getBooleanProperty should be used to return configuration parameters from the underlying system.
abstract Config	getConfig() Method getConfig returns the actual instance of the underlying configuration instance.
abstract Config	getConfigCopy()
abstract long	getCounterValue(Enum counter) Method getCounterValue is used to retrieve a counter value.
abstract long	getCounterValue(String group, String counter) Method getCounterValue is used to retrieve a counter value.
FlowSession	getCurrentSession() Method getCurrentSession returns the currentSession of this FlowProcess object.
abstract int	getCurrentSliceNum() Method getCurrentSliceNum returns an integer representing which slice instance currently running.
String	getID() Method getID() returns the current
Integer	getIntegerProperty(String key) Method getIntegerProperty should be used to return configuration parameters from the underlying system.
int	getIntegerProperty(String key, int defaultValue) Method getIntegerProperty should be used to return configuration parameters from the underlying system.
abstract int	getNumProcessSlices() Method getNumProcessSlices returns the number of parallel slices or tasks allocated for this process execution.
abstract Object	getProperty(String key) Method getProperty should be used to return configuration parameters from the underlying system.
abstract Collection<String>	getPropertyKeys() Method getPropertyKeys returns an immutable collection of all available property key values.
String	getStringProperty(String key) Method getStringProperty should be used to return configuration parameters from the underlying system.
String	getStringProperty(String key, String defaultValue) Method getStringProperty should be used to return configuration parameters from the underlying system.
TupleEntryCollector	getTrapCollectorFor(Tap trap) Method getTrapCollectorFor will return a new TupleEntryCollector if one hasn't previously been created for the given trap Tap.
protected Map<Tap,TupleEntryCollector>	getTrapCollectors()
abstract void	increment(Enum counter, long amount) Method increment is used to increment a custom counter.
abstract void	increment(String group, String counter, long amount) Method increment is used to increment a custom counter.
abstract boolean	isCounterStatusInitialized() Method isCounterStatusInitialized returns true if it is safe to increment a counter or set a status.
abstract void	keepAlive() Method keepAlive notifies the system that the current process is still alive.
abstract Config	mergeMapIntoConfig(Config defaultConfig, Map<String,String> map)
abstract Object	newInstance(String className) Method newInstance creates a new object instance from the given className argument delegating to any platform specific instantiation and configuration routines.
abstract TupleEntryCollector	openSystemIntermediateForWrite()
abstract TupleEntryIterator	openTapForRead(Tap tap) Method openTapForRead return a TupleEntryIterator for the given Tap instance.
abstract TupleEntryCollector	openTapForWrite(Tap tap) Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.
abstract TupleEntryCollector	openTrapForWrite(Tap trap) Method openTrapForWrite returns a (@link TupleCollector} for the given Tap instance.
void	setCurrentSession(FlowSession currentSession) Method setCurrentSession sets the currentSession of this FlowProcess object.
abstract void	setStatus(String status) Method setStatus is used to set the status of the current operation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NULL

public static FlowProcess NULL

Field NULL is a noop implementation of FlowSession.

Constructor Detail

FlowProcess

protected FlowProcess()

FlowProcess

protected FlowProcess(FlowSession currentSession)

FlowProcess

protected FlowProcess(FlowProcess<Config> flowProcess)

Copy constructor.

Shares the underlying trap collector collection across copies to avoid a static collection.

Parameters:

flowProcess -

Method Detail

copyWith

public abstract FlowProcess<Config> copyWith(Config config)

getID

public String getID()

Method getID() returns the current

Returns:

of type String

getCurrentSession

public FlowSession getCurrentSession()

Method getCurrentSession returns the currentSession of this FlowProcess object.

Returns:

the currentSession (type FlowSession) of this FlowProcess object.

setCurrentSession

public void setCurrentSession(FlowSession currentSession)

Method setCurrentSession sets the currentSession of this FlowProcess object.

Parameters:

currentSession - the currentSession of this FlowProcess object.

getNumProcessSlices

public abstract int getNumProcessSlices()

Method getNumProcessSlices returns the number of parallel slices or tasks allocated for this process execution.

For MapReduce platforms, this is the same as the number of tasks for a given MapReduce job.

Returns:

an int

getCurrentSliceNum

public abstract int getCurrentSliceNum()

Method getCurrentSliceNum returns an integer representing which slice instance currently running.

0 (zero) is the first slice instance.

Returns:

an int

getProperty

public abstract Object getProperty(String key)

Method getProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type String

Returns:

an Object

getStringProperty

public String getStringProperty(String key)

Method getStringProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type String,

Returns:

null if property is not set

getStringProperty

public String getStringProperty(String key,
                       String defaultValue)

Method getStringProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type String,

defaultValue - of type String,

Returns:

defaultValue if property is not set

getIntegerProperty

public Integer getIntegerProperty(String key)

Method getIntegerProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type String,

Returns:

null if property is not set

getIntegerProperty

public int getIntegerProperty(String key,
                     int defaultValue)

Method getIntegerProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type String,

defaultValue - of type int,

Returns:

defaultValue if property is not set

getBooleanProperty

public Boolean getBooleanProperty(String key)

Method getBooleanProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type Boolean, null if property is not set

Returns:

an Object

getBooleanProperty

public boolean getBooleanProperty(String key,
                         boolean defaultValue)

Method getBooleanProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current Configuration will be queried.

Parameters:

key - of type String

defaultValue - of type boolean

Returns:

defaultValue if property is not set

getPropertyKeys

public abstract Collection<String> getPropertyKeys()

Method getPropertyKeys returns an immutable collection of all available property key values.

Returns:

a Collection

newInstance

public abstract Object newInstance(String className)

Method newInstance creates a new object instance from the given className argument delegating to any platform specific instantiation and configuration routines.

Parameters:

className -

Returns:

an instance of className

keepAlive

public abstract void keepAlive()

Method keepAlive notifies the system that the current process is still alive. Use this method if a particular Operation takes some moments to complete. Each system is different, so calling ping every few seconds to every minute or so would be best.

This method will fail silently if the underlying mechanism to notify keepAlive status are not initialized.

increment

public abstract void increment(Enum counter,
             long amount)

Method increment is used to increment a custom counter. Counters must be of type Enum. The amount to increment must be a integer value.

This method will fail if the underlying counter infrastructure is unavailable. See isCounterStatusInitialized().

Parameters:

counter - of type Enum

amount - of type int

increment

public abstract void increment(String group,
             String counter,
             long amount)

Method increment is used to increment a custom counter. The amount to increment must be a integer value.

This method will fail if the underlying counter infrastructure is unavailable. See isCounterStatusInitialized().

Parameters:

group - of type String

counter - of type String

amount - of type int

getCounterValue

public abstract long getCounterValue(Enum counter)

Method getCounterValue is used to retrieve a counter value.

This method will fail if the underlying counter infrastructure is unavailable. See isCounterStatusInitialized().

Parameters:

counter - of type Enum

getCounterValue

public abstract long getCounterValue(String group,
                   String counter)

Method getCounterValue is used to retrieve a counter value.

This method will fail if the underlying counter infrastructure is unavailable. See isCounterStatusInitialized().

Parameters:

group - of type String

counter - of type String

setStatus

public abstract void setStatus(String status)

Method setStatus is used to set the status of the current operation.

This method will fail if the underlying counter infrastructure is unavailable. See isCounterStatusInitialized().

Parameters:

status - of type String

isCounterStatusInitialized

public abstract boolean isCounterStatusInitialized()

Method isCounterStatusInitialized returns true if it is safe to increment a counter or set a status.

Returns:

boolean

openTapForRead

public abstract TupleEntryIterator openTapForRead(Tap tap)
                                           throws IOException

Method openTapForRead return a TupleEntryIterator for the given Tap instance.

Note the returned iterator will return the same instance of TupleEntry on every call, thus a copy must be made of either the TupleEntry or the underlying Tuple instance if they are to be stored in a Collection.

Parameters:

tap - of type Tap

Returns:

TupleIterator

Throws:

IOException - when there is a failure opening the resource

openTapForWrite

public abstract TupleEntryCollector openTapForWrite(Tap tap)
                                             throws IOException

Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.

Parameters:

tap - of type Tap

Returns:

TupleCollector

Throws:

IOException - when there is a failure opening the resource

openTrapForWrite

public abstract TupleEntryCollector openTrapForWrite(Tap trap)
                                              throws IOException

Method openTrapForWrite returns a (@link TupleCollector} for the given Tap instance.

Parameters:

trap - of type Tap

Returns:

TupleCollector

Throws:

IOException - when there is a failure opening the resource

openSystemIntermediateForWrite

public abstract TupleEntryCollector openSystemIntermediateForWrite()
                                                            throws IOException

Throws:

IOException

getConfig

public abstract Config getConfig()

Method getConfig returns the actual instance of the underlying configuration instance.

This instance should not be modified or cached, see getConfigCopy() for a modifiable instance.

Returns:

an instance of the configuration

getConfigCopy

public abstract Config getConfigCopy()

copyConfig

public abstract <C> C copyConfig(C config)

diffConfigIntoMap

public abstract <C> Map<String,String> diffConfigIntoMap(C defaultConfig,
                                       C updatedConfig)

mergeMapIntoConfig

public abstract Config mergeMapIntoConfig(Config defaultConfig,
                        Map<String,String> map)

getTrapCollectorFor

public TupleEntryCollector getTrapCollectorFor(Tap trap)

Method getTrapCollectorFor will return a new TupleEntryCollector if one hasn't previously been created for the given trap Tap.

Parameters:

trap -

Returns:

TupleEntryCollector

getTrapCollectors

protected Map<Tap,TupleEntryCollector> getTrapCollectors()

closeTrapCollectors

public void closeTrapCollectors()