|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.flow.FlowProcess<JobConf> cascading.flow.hadoop.HadoopFlowProcess
public class HadoopFlowProcess
Class HadoopFlowProcess is an implementation of FlowProcess
for Hadoop. Use this interface to get direct
access to the Hadoop JobConf and Reporter interfaces.
Operation
s to
fail if they are executed on a system other than Hadoop.
FlowSession
,
JobConf
,
Reporter
Nested Class Summary |
---|
Nested classes/interfaces inherited from class cascading.flow.FlowProcess |
---|
FlowProcess.NullFlowProcess |
Field Summary |
---|
Fields inherited from class cascading.flow.FlowProcess |
---|
NULL |
Constructor Summary | |
---|---|
HadoopFlowProcess()
|
|
HadoopFlowProcess(FlowSession flowSession,
JobConf jobConf)
|
|
HadoopFlowProcess(FlowSession flowSession,
JobConf jobConf,
boolean isMapper)
Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance. |
|
HadoopFlowProcess(HadoopFlowProcess flowProcess,
JobConf jobConf)
|
|
HadoopFlowProcess(JobConf jobConf)
|
Method Summary | |
---|---|
JobConf |
copyConfig(JobConf jobConf)
|
FlowProcess |
copyWith(JobConf jobConf)
|
Map<String,String> |
diffConfigIntoMap(JobConf defaultConfig,
JobConf updatedConfig)
|
JobConf |
getConfigCopy()
|
int |
getCurrentNumMappers()
|
int |
getCurrentNumReducers()
|
int |
getCurrentSliceNum()
Method getCurrentTaskNum returns the task number of this task. |
JobConf |
getJobConf()
Method getJobConf returns the jobConf of this HadoopFlowProcess object. |
int |
getNumProcessSlices()
Method getNumProcessSlices returns the number of parallel slices or tasks allocated for this process execution. |
OutputCollector |
getOutputCollector()
|
Object |
getProperty(String key)
Method getProperty should be used to return configuration parameters from the underlying system. |
Collection<String> |
getPropertyKeys()
Method getPropertyKeys returns an immutable collection of all available property key values. |
Reporter |
getReporter()
Method getReporter returns the reporter of this HadoopFlowProcess object. |
void |
increment(Enum counter,
long amount)
Method increment is used to increment a custom counter. |
void |
increment(String group,
String counter,
long amount)
Method increment is used to increment a custom counter. |
boolean |
isCounterStatusInitialized()
Method isCounterStatusInitialized returns true if it is safe to increment a counter or set a status. |
boolean |
isMapper()
Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper. |
void |
keepAlive()
Method keepAlive notifies the system that the current process is still alive. |
JobConf |
mergeMapIntoConfig(JobConf defaultConfig,
Map<String,String> map)
|
Object |
newInstance(String className)
Method newInstance creates a new object instance from the given className argument delegating to any platform specific instantiation and configuration routines. |
TupleEntryCollector |
openSystemIntermediateForWrite()
|
TupleEntryIterator |
openTapForRead(Tap tap)
Method openTapForRead return a TupleEntryIterator for the given Tap instance. |
TupleEntryCollector |
openTapForWrite(Tap tap)
Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance. |
TupleEntryCollector |
openTrapForWrite(Tap trap)
Method openTrapForWrite returns a (@link TupleCollector} for the given Tap instance. |
void |
setOutputCollector(OutputCollector outputCollector)
|
void |
setReporter(Reporter reporter)
Method setReporter sets the reporter of this HadoopFlowProcess object. |
void |
setStatus(String status)
Method setStatus is used to set the status of the current operation. |
Methods inherited from class cascading.flow.FlowProcess |
---|
getCurrentSession, getID, getStringProperty, setCurrentSession |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public HadoopFlowProcess()
public HadoopFlowProcess(JobConf jobConf)
public HadoopFlowProcess(FlowSession flowSession, JobConf jobConf)
public HadoopFlowProcess(FlowSession flowSession, JobConf jobConf, boolean isMapper)
flowSession
- of type FlowSessionjobConf
- of type JobConfpublic HadoopFlowProcess(HadoopFlowProcess flowProcess, JobConf jobConf)
Method Detail |
---|
public FlowProcess copyWith(JobConf jobConf)
copyWith
in class FlowProcess<JobConf>
public JobConf getJobConf()
public JobConf getConfigCopy()
getConfigCopy
in class FlowProcess<JobConf>
public boolean isMapper()
public int getCurrentNumMappers()
public int getCurrentNumReducers()
public int getCurrentSliceNum()
getCurrentSliceNum
in class FlowProcess<JobConf>
public int getNumProcessSlices()
FlowProcess
getNumProcessSlices
in class FlowProcess<JobConf>
public void setReporter(Reporter reporter)
reporter
- the reporter of this HadoopFlowProcess object.public Reporter getReporter()
public void setOutputCollector(OutputCollector outputCollector)
public OutputCollector getOutputCollector()
public Object getProperty(String key)
FlowProcess
getProperty
in class FlowProcess<JobConf>
key
- of type String
public Collection<String> getPropertyKeys()
FlowProcess
getPropertyKeys
in class FlowProcess<JobConf>
public Object newInstance(String className)
FlowProcess
newInstance
in class FlowProcess<JobConf>
public void keepAlive()
FlowProcess
Operation
takes some moments to complete. Each system is different, so calling
ping every few seconds to every minute or so would be best.
This method will fail silently if the underlying mechanism to notify keepAlive status are not initialized.
keepAlive
in class FlowProcess<JobConf>
public void increment(Enum counter, long amount)
FlowProcess
FlowProcess.isCounterStatusInitialized()
.
increment
in class FlowProcess<JobConf>
counter
- of type Enumamount
- of type intpublic void increment(String group, String counter, long amount)
FlowProcess
FlowProcess.isCounterStatusInitialized()
.
increment
in class FlowProcess<JobConf>
group
- of type Stringcounter
- of type Stringamount
- of type intpublic void setStatus(String status)
FlowProcess
FlowProcess.isCounterStatusInitialized()
.
setStatus
in class FlowProcess<JobConf>
status
- of type Stringpublic boolean isCounterStatusInitialized()
FlowProcess
isCounterStatusInitialized
in class FlowProcess<JobConf>
public TupleEntryIterator openTapForRead(Tap tap) throws IOException
FlowProcess
TupleEntryIterator
for the given Tap instance.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
openTapForRead
in class FlowProcess<JobConf>
tap
- of type Tap
IOException
- when there is a failure opening the resourcepublic TupleEntryCollector openTapForWrite(Tap tap) throws IOException
FlowProcess
openTapForWrite
in class FlowProcess<JobConf>
tap
- of type Tap
IOException
- when there is a failure opening the resourcepublic TupleEntryCollector openTrapForWrite(Tap trap) throws IOException
FlowProcess
openTrapForWrite
in class FlowProcess<JobConf>
trap
- of type Tap
IOException
- when there is a failure opening the resourcepublic TupleEntryCollector openSystemIntermediateForWrite() throws IOException
openSystemIntermediateForWrite
in class FlowProcess<JobConf>
IOException
public JobConf copyConfig(JobConf jobConf)
copyConfig
in class FlowProcess<JobConf>
public Map<String,String> diffConfigIntoMap(JobConf defaultConfig, JobConf updatedConfig)
diffConfigIntoMap
in class FlowProcess<JobConf>
public JobConf mergeMapIntoConfig(JobConf defaultConfig, Map<String,String> map)
mergeMapIntoConfig
in class FlowProcess<JobConf>
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |