cascading.flow.hadoop
Class HadoopFlowProcess

java.lang.Object
  extended by cascading.flow.FlowProcess<JobConf>
      extended by cascading.flow.hadoop.HadoopFlowProcess

public class HadoopFlowProcess
extends cascading.flow.FlowProcess<JobConf>

Class HadoopFlowProcess is an implementation of FlowProcess for Hadoop. Use this interface to get direct access to the Hadoop JobConf and Reporter interfaces.

Be warned that coupling to this implementation will cause custom Operations to fail if they are executed on a system other than Hadoop.

See Also:
FlowSession, JobConf, Reporter

Nested Class Summary
 
Nested classes/interfaces inherited from class cascading.flow.FlowProcess
cascading.flow.FlowProcess.NullFlowProcess
 
Field Summary
 
Fields inherited from class cascading.flow.FlowProcess
NULL
 
Constructor Summary
HadoopFlowProcess()
           
HadoopFlowProcess(cascading.flow.FlowSession flowSession, JobConf jobConf)
           
HadoopFlowProcess(cascading.flow.FlowSession flowSession, JobConf jobConf, boolean isMapper)
          Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.
HadoopFlowProcess(HadoopFlowProcess flowProcess, JobConf jobConf)
           
HadoopFlowProcess(JobConf jobConf)
           
 
Method Summary
 JobConf copyConfig(JobConf jobConf)
           
 cascading.flow.FlowProcess copyWith(JobConf jobConf)
           
 Map<String,String> diffConfigIntoMap(JobConf defaultConfig, JobConf updatedConfig)
           
 JobConf getConfigCopy()
           
 int getCurrentNumMappers()
           
 int getCurrentNumReducers()
           
 int getCurrentSliceNum()
          Method getCurrentTaskNum returns the task number of this task.
 JobConf getJobConf()
          Method getJobConf returns the jobConf of this HadoopFlowProcess object.
 int getNumProcessSlices()
           
 OutputCollector getOutputCollector()
           
 Object getProperty(String key)
           
 Collection<String> getPropertyKeys()
           
 Reporter getReporter()
          Method getReporter returns the reporter of this HadoopFlowProcess object.
 void increment(Enum counter, long amount)
           
 void increment(String group, String counter, long amount)
           
 boolean isCounterStatusInitialized()
           
 boolean isMapper()
          Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper.
 void keepAlive()
           
 JobConf mergeMapIntoConfig(JobConf defaultConfig, Map<String,String> map)
           
 Object newInstance(String className)
           
 cascading.tuple.TupleEntryCollector openSystemIntermediateForWrite()
           
 cascading.tuple.TupleEntryIterator openTapForRead(cascading.tap.Tap tap)
           
 cascading.tuple.TupleEntryCollector openTapForWrite(cascading.tap.Tap tap)
           
 cascading.tuple.TupleEntryCollector openTrapForWrite(cascading.tap.Tap trap)
           
 void setOutputCollector(OutputCollector outputCollector)
           
 void setReporter(Reporter reporter)
          Method setReporter sets the reporter of this HadoopFlowProcess object.
 void setStatus(String status)
           
 
Methods inherited from class cascading.flow.FlowProcess
getCurrentSession, getID, getIntegerProperty, getStringProperty, setCurrentSession
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

HadoopFlowProcess

public HadoopFlowProcess()

HadoopFlowProcess

public HadoopFlowProcess(JobConf jobConf)

HadoopFlowProcess

public HadoopFlowProcess(cascading.flow.FlowSession flowSession,
                         JobConf jobConf)

HadoopFlowProcess

public HadoopFlowProcess(cascading.flow.FlowSession flowSession,
                         JobConf jobConf,
                         boolean isMapper)
Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

Parameters:
flowSession - of type FlowSession
jobConf - of type JobConf

HadoopFlowProcess

public HadoopFlowProcess(HadoopFlowProcess flowProcess,
                         JobConf jobConf)
Method Detail

copyWith

public cascading.flow.FlowProcess copyWith(JobConf jobConf)
Specified by:
copyWith in class cascading.flow.FlowProcess<JobConf>

getJobConf

public JobConf getJobConf()
Method getJobConf returns the jobConf of this HadoopFlowProcess object.

Returns:
the jobConf (type JobConf) of this HadoopFlowProcess object.

getConfigCopy

public JobConf getConfigCopy()
Specified by:
getConfigCopy in class cascading.flow.FlowProcess<JobConf>

isMapper

public boolean isMapper()
Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper. If false, it is a reducer.

Returns:
boolean

getCurrentNumMappers

public int getCurrentNumMappers()

getCurrentNumReducers

public int getCurrentNumReducers()

getCurrentSliceNum

public int getCurrentSliceNum()
Method getCurrentTaskNum returns the task number of this task. Task 0 is the first task.

Specified by:
getCurrentSliceNum in class cascading.flow.FlowProcess<JobConf>
Returns:
int

getNumProcessSlices

public int getNumProcessSlices()
Specified by:
getNumProcessSlices in class cascading.flow.FlowProcess<JobConf>

setReporter

public void setReporter(Reporter reporter)
Method setReporter sets the reporter of this HadoopFlowProcess object.

Parameters:
reporter - the reporter of this HadoopFlowProcess object.

getReporter

public Reporter getReporter()
Method getReporter returns the reporter of this HadoopFlowProcess object.

Returns:
the reporter (type Reporter) of this HadoopFlowProcess object.

setOutputCollector

public void setOutputCollector(OutputCollector outputCollector)

getOutputCollector

public OutputCollector getOutputCollector()

getProperty

public Object getProperty(String key)
Specified by:
getProperty in class cascading.flow.FlowProcess<JobConf>

getPropertyKeys

public Collection<String> getPropertyKeys()
Specified by:
getPropertyKeys in class cascading.flow.FlowProcess<JobConf>

newInstance

public Object newInstance(String className)
Specified by:
newInstance in class cascading.flow.FlowProcess<JobConf>

keepAlive

public void keepAlive()
Specified by:
keepAlive in class cascading.flow.FlowProcess<JobConf>

increment

public void increment(Enum counter,
                      long amount)
Specified by:
increment in class cascading.flow.FlowProcess<JobConf>

increment

public void increment(String group,
                      String counter,
                      long amount)
Specified by:
increment in class cascading.flow.FlowProcess<JobConf>

setStatus

public void setStatus(String status)
Specified by:
setStatus in class cascading.flow.FlowProcess<JobConf>

isCounterStatusInitialized

public boolean isCounterStatusInitialized()
Specified by:
isCounterStatusInitialized in class cascading.flow.FlowProcess<JobConf>

openTapForRead

public cascading.tuple.TupleEntryIterator openTapForRead(cascading.tap.Tap tap)
                                                  throws IOException
Specified by:
openTapForRead in class cascading.flow.FlowProcess<JobConf>
Throws:
IOException

openTapForWrite

public cascading.tuple.TupleEntryCollector openTapForWrite(cascading.tap.Tap tap)
                                                    throws IOException
Specified by:
openTapForWrite in class cascading.flow.FlowProcess<JobConf>
Throws:
IOException

openTrapForWrite

public cascading.tuple.TupleEntryCollector openTrapForWrite(cascading.tap.Tap trap)
                                                     throws IOException
Specified by:
openTrapForWrite in class cascading.flow.FlowProcess<JobConf>
Throws:
IOException

openSystemIntermediateForWrite

public cascading.tuple.TupleEntryCollector openSystemIntermediateForWrite()
                                                                   throws IOException
Specified by:
openSystemIntermediateForWrite in class cascading.flow.FlowProcess<JobConf>
Throws:
IOException

copyConfig

public JobConf copyConfig(JobConf jobConf)
Specified by:
copyConfig in class cascading.flow.FlowProcess<JobConf>

diffConfigIntoMap

public Map<String,String> diffConfigIntoMap(JobConf defaultConfig,
                                            JobConf updatedConfig)
Specified by:
diffConfigIntoMap in class cascading.flow.FlowProcess<JobConf>

mergeMapIntoConfig

public JobConf mergeMapIntoConfig(JobConf defaultConfig,
                                  Map<String,String> map)
Specified by:
mergeMapIntoConfig in class cascading.flow.FlowProcess<JobConf>


Copyright © 2007-2013 Concurrent, Inc. All Rights Reserved.