cascading.flow.hadoop
Class HadoopFlowProcess

java.lang.Object
  cascading.flow.FlowProcess
      cascading.flow.hadoop.HadoopFlowProcess

public class HadoopFlowProcess
extends FlowProcess

Class HadoopFlowProcess is an implemenation of FlowProcess for Hadoop. Use this interfact to get direct access to the Hadoop JobConf and Reporter interfaces.

Be warned that coupling to this implemenation will cause custom Operations to fail if they are executed on a system other than Hadoop.

See Also:

FlowSession, JobConf, Reporter

Field Summary

Fields inherited from class cascading.flow.FlowProcess

NULL

Constructor Summary

HadoopFlowProcess(FlowSession flowSession, JobConf jobConf, boolean isMapper)
Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

Method Summary

int	getCurrentNumMappers()
int	getCurrentNumReducers()
int	getCurrentTaskNum() Method getCurrentTaskNum returns the task number of this task.
JobConf	getJobConf() Method getJobConf returns the jobConf of this HadoopFlowProcess object.
Object	getProperty(String key) Method getProperty should be used to return configuration parameters from the underlying system.
Reporter	getReporter() Method getReporter returns the reporter of this HadoopFlowProcess object.
void	increment(Enum counter, int amount) Method increment is used to increment a custom counter.
void	increment(String group, String counter, int amount) Method increment is used to increment a custom counter.
boolean	isMapper() Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper.
void	keepAlive() Method keepAlive notifies the system that the current process is still alive.
TupleEntryIterator	openTapForRead(Tap tap) Method openTapForRead return a TupleIterator for the given Tap instance.
TupleEntryCollector	openTapForWrite(Tap tap) Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.
void	setReporter(Reporter reporter) Method setReporter sets the reporter of this HadoopFlowProcess object.
void	setStatus(String status) Method setStatus is used to set the status of the current operation.

Methods inherited from class cascading.flow.FlowProcess

getCurrentSession, setCurrentSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFlowProcess

public HadoopFlowProcess(FlowSession flowSession,
                         JobConf jobConf,
                         boolean isMapper)

Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

Parameters:

flowSession - of type FlowSession

jobConf - of type JobConf

Method Detail

getJobConf

public JobConf getJobConf()

Method getJobConf returns the jobConf of this HadoopFlowProcess object.

Returns:

the jobConf (type JobConf) of this HadoopFlowProcess object.

isMapper

public boolean isMapper()

Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper. If false, it is a reducer.

Returns:

boolean

getCurrentNumMappers

public int getCurrentNumMappers()

getCurrentNumReducers

public int getCurrentNumReducers()

getCurrentTaskNum

public int getCurrentTaskNum()

Method getCurrentTaskNum returns the task number of this task. Task 0 is the first task.

Returns:

int

setReporter

public void setReporter(Reporter reporter)

Method setReporter sets the reporter of this HadoopFlowProcess object.

Parameters:

reporter - the reporter of this HadoopFlowProcess object.

getReporter

public Reporter getReporter()

Method getReporter returns the reporter of this HadoopFlowProcess object.

Returns:

the reporter (type Reporter) of this HadoopFlowProcess object.

getProperty

public Object getProperty(String key)

Description copied from class: FlowProcess

Method getProperty should be used to return configuration parameters from the underlying system.

In the case of Hadoop, the current JobConf will be queried.

Specified by:

getProperty in class FlowProcess

Parameters:

key - of type String

Returns:

an Object

See Also:

FlowProcess.getProperty(String)

keepAlive

public void keepAlive()

Description copied from class: FlowProcess

Method keepAlive notifies the system that the current process is still alive. Use this method if a particular Operation takes some moments to complete. Each system is different, so calling ping every few seconds to every minute or so would be best.

Specified by:

keepAlive in class FlowProcess

See Also:

FlowProcess.keepAlive()

increment

public void increment(Enum counter,
                      int amount)

Description copied from class: FlowProcess

Method increment is used to increment a custom counter. Counters must be of type Enum. The amount to increment must be a integer value.

Specified by:

increment in class FlowProcess

Parameters:

counter - of type Enum

amount - of type int

See Also:

FlowProcess.increment(Enum, int)

increment

public void increment(String group,
                      String counter,
                      int amount)

Description copied from class: FlowProcess

Method increment is used to increment a custom counter. The amount to increment must be a integer value.

Specified by:

increment in class FlowProcess

Parameters:

group - of type String

counter - of type String

amount - of type int

See Also:

FlowProcess.increment(String, String, int)

setStatus

public void setStatus(String status)

Description copied from class: FlowProcess

Method setStatus is used to set the status of the current operation.

Specified by:

setStatus in class FlowProcess

Parameters:

status - of type String

See Also:

FlowProcess.setStatus(String)

openTapForRead

public TupleEntryIterator openTapForRead(Tap tap)
                                  throws IOException

Description copied from class: FlowProcess

Method openTapForRead return a TupleIterator for the given Tap instance.

Specified by:

openTapForRead in class FlowProcess

Parameters:

tap - of type Tap

Returns:

TupleIterator

Throws:

IOException - when there is a failure opening the resource

See Also:

FlowProcess.openTapForRead(Tap)

openTapForWrite

public TupleEntryCollector openTapForWrite(Tap tap)
                                    throws IOException

Description copied from class: FlowProcess

Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.

Specified by:

openTapForWrite in class FlowProcess

Parameters:

tap - of type Tap

Returns:

TupleCollector

Throws:

IOException - when there is a failure opening the resource

See Also:

FlowProcess.openTapForWrite(Tap)