cascading.stats.hadoop
Class BaseHadoopStepStats

java.lang.Object
  extended by cascading.stats.CascadingStats
      extended by cascading.stats.FlowStepStats
          extended by cascading.stats.hadoop.BaseHadoopStepStats
All Implemented Interfaces:
Serializable
Direct Known Subclasses:
HadoopStepStats

public abstract class BaseHadoopStepStats
extends FlowStepStats

Class BaseHadoopStepStats is a base class to Hadoop specific statistics and methods to underlying Hadoop facilities.

See Also:
Serialized Form

Nested Class Summary
 
Nested classes/interfaces inherited from class cascading.stats.CascadingStats
CascadingStats.Status
 
Field Summary
static String COUNTER_TIMEOUT_PROPERTY
           
static int TIMEOUT_MAX
           
 
Fields inherited from class cascading.stats.CascadingStats
clientState, STATS_STORE_INTERVAL
 
Constructor Summary
protected BaseHadoopStepStats(FlowStep<JobConf> flowStep, ClientState clientState)
           
 
Method Summary
protected abstract  void addAttemptsToTaskStats(Map<String,HadoopSliceStats> taskStats, boolean captureAttempts)
           
protected abstract  void addTaskStats(Map<String,HadoopSliceStats> taskStats, HadoopSliceStats.Kind kind, boolean skipLast)
           
protected  Counters cachedCounters()
           
protected  Counters cachedCounters(boolean force)
           
 void captureDetail()
          Method captureDetail captures statistics task details and completion events.
 void captureDetail(boolean captureAttempts)
           
 Set<String> getChildIDs()
           
 Collection getChildren()
          Method getChildren returns the children of this HadoopStepStats object.
 Collection<String> getCounterGroups()
          Method getCounterGroups returns all of the Hadoop counter groups.
 Collection<String> getCounterGroupsMatching(String regex)
          Method getCounterGroupsMatching returns all the Hadoop counter groups that match the give regex pattern.
 Collection<String> getCountersFor(String group)
          Method getCountersFor returns the Hadoop counters for the given group.
 long getCounterValue(Enum counter)
          Method getCounterValue returns the Hadoop counter value for the given counter enum.
 long getCounterValue(String group, String counter)
          Method getCounterValue returns the Hadoop counter value for the given group and counter name.
abstract  JobClient getJobClient()
          Method getJobClient returns the Hadoop JobClient managing this Hadoop job.
 String getJobID()
          Method getJobID returns the Hadoop running job JobID.
 float getMapProgress()
          Returns the underlying Map tasks progress percentage.
 int getNumMapTasks()
          Method getNumMapTasks returns the numMapTasks from the Hadoop job file.
 int getNumReduceTasks()
          Method getNumReduceTasks returns the numReducerTasks from the Hadoop job file.
 float getReduceProgress()
          Returns the underlying Reduce tasks progress percentage.
abstract  RunningJob getRunningJob()
          Method getRunningJob returns the Hadoop RunningJob managing this Hadoop job.
 String getStatusURL()
           
 Map<String,HadoopSliceStats> getTaskStats()
          Method getTaskStats returns the taskStats of this HadoopStepStats object.
 void recordChildStats()
          Synchronized to prevent state changes mid record, #stop may be called out of band
protected  void setTaskStats(Map<String,HadoopSliceStats> taskStats)
           
 
Methods inherited from class cascading.stats.FlowStepStats
getFlowStep, getID, recordInfo, toString
 
Methods inherited from class cascading.stats.CascadingStats
cleanup, getCountersFor, getCurrentDuration, getDuration, getFinishedTime, getName, getPendingTime, getRunTime, getStartTime, getStatsString, getStatus, getSubmitTime, getThrowable, isEngaged, isFailed, isFinished, isPending, isRunning, isSkipped, isStarted, isStopped, isSubmitted, isSuccessful, markFailed, markPending, markPendingTime, markRunning, markRunTime, markSkipped, markStarted, markStartedThenRunning, markStartTime, markStartToRunTime, markStopped, markSubmitted, markSubmitTime, markSuccessful, prepare, recordStats, setStatsStoreInterval
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

COUNTER_TIMEOUT_PROPERTY

public static final String COUNTER_TIMEOUT_PROPERTY
See Also:
Constant Field Values

TIMEOUT_MAX

public static final int TIMEOUT_MAX
See Also:
Constant Field Values
Constructor Detail

BaseHadoopStepStats

protected BaseHadoopStepStats(FlowStep<JobConf> flowStep,
                              ClientState clientState)
Method Detail

getTaskStats

public Map<String,HadoopSliceStats> getTaskStats()
Method getTaskStats returns the taskStats of this HadoopStepStats object.

Returns:
the taskStats (type ArrayList) of this HadoopStepStats object.

setTaskStats

protected void setTaskStats(Map<String,HadoopSliceStats> taskStats)

getNumMapTasks

public int getNumMapTasks()
Method getNumMapTasks returns the numMapTasks from the Hadoop job file.

Returns:
the numMapTasks (type int) of this HadoopStepStats object.

getNumReduceTasks

public int getNumReduceTasks()
Method getNumReduceTasks returns the numReducerTasks from the Hadoop job file.

Returns:
the numReducerTasks (type int) of this HadoopStepStats object.

getJobID

public String getJobID()
Method getJobID returns the Hadoop running job JobID.

Returns:
the jobID (type String) of this HadoopStepStats object.

getJobClient

public abstract JobClient getJobClient()
Method getJobClient returns the Hadoop JobClient managing this Hadoop job.

Returns:
the jobClient (type JobClient) of this HadoopStepStats object.

getRunningJob

public abstract RunningJob getRunningJob()
Method getRunningJob returns the Hadoop RunningJob managing this Hadoop job.

Returns:
the runningJob (type RunningJob) of this HadoopStepStats object.

getCounterGroups

public Collection<String> getCounterGroups()
Method getCounterGroups returns all of the Hadoop counter groups.

Specified by:
getCounterGroups in class CascadingStats
Returns:
the counterGroups (type Collection) of this HadoopStepStats object.

getCounterGroupsMatching

public Collection<String> getCounterGroupsMatching(String regex)
Method getCounterGroupsMatching returns all the Hadoop counter groups that match the give regex pattern.

Specified by:
getCounterGroupsMatching in class CascadingStats
Parameters:
regex - of String
Returns:
Collection

getCountersFor

public Collection<String> getCountersFor(String group)
Method getCountersFor returns the Hadoop counters for the given group.

Specified by:
getCountersFor in class CascadingStats
Parameters:
group - of String
Returns:
Collection

getCounterValue

public long getCounterValue(Enum counter)
Method getCounterValue returns the Hadoop counter value for the given counter enum.

Specified by:
getCounterValue in class CascadingStats
Parameters:
counter - of Enum
Returns:
long

getCounterValue

public long getCounterValue(String group,
                            String counter)
Method getCounterValue returns the Hadoop counter value for the given group and counter name.

Specified by:
getCounterValue in class CascadingStats
Parameters:
group - of String
counter - of String
Returns:
long

cachedCounters

protected Counters cachedCounters()

cachedCounters

protected Counters cachedCounters(boolean force)

getMapProgress

public float getMapProgress()
Returns the underlying Map tasks progress percentage.

This method is experimental.

Returns:
float

getReduceProgress

public float getReduceProgress()
Returns the underlying Reduce tasks progress percentage.

This method is experimental.

Returns:
float

getStatusURL

public String getStatusURL()

getChildren

public Collection getChildren()
Method getChildren returns the children of this HadoopStepStats object.

Specified by:
getChildren in class CascadingStats
Returns:
the children (type Collection) of this HadoopStepStats object.

getChildIDs

public Set<String> getChildIDs()

recordChildStats

public void recordChildStats()
Synchronized to prevent state changes mid record, #stop may be called out of band

Specified by:
recordChildStats in class FlowStepStats

captureDetail

public void captureDetail()
Method captureDetail captures statistics task details and completion events.

Specified by:
captureDetail in class CascadingStats

captureDetail

public void captureDetail(boolean captureAttempts)

addTaskStats

protected abstract void addTaskStats(Map<String,HadoopSliceStats> taskStats,
                                     HadoopSliceStats.Kind kind,
                                     boolean skipLast)
                              throws IOException
Throws:
IOException

addAttemptsToTaskStats

protected abstract void addAttemptsToTaskStats(Map<String,HadoopSliceStats> taskStats,
                                               boolean captureAttempts)


Copyright © 2007-2014 Concurrent, Inc. All Rights Reserved.