cascading.platform.hadoop
Class HadoopPlatform

java.lang.Object
  cascading.platform.TestPlatform
      cascading.platform.hadoop.BaseHadoopPlatform
          cascading.platform.hadoop.HadoopPlatform

public class HadoopPlatform
extends BaseHadoopPlatform

Class HadoopPlatform is automatically loaded and injected into a PlatformTestCase instance so that all *PlatformTest classes can be tested against Apache Hadoop.

This platform works in three modes.

Hadoop standalone mode is when Hadoop is NOT run as a cluster, and all child tasks are in process and in memory of the "client" side code.

Hadoop mini cluster mode where a cluster is created on demand using the Hadoop MiniDFSCluster and MiniMRCluster utilities. When a PlatformTestCase requests to use a cluster, this is the default cluster. All properties are pulled from the current CLASSPATH via the JobConf.

Lastly remote cluster mode is enabled when the System property "mapred.jar" is set. This is a Hadoop property specifying the Hadoop "job jar" to be used cluster side. This MUST be the Cascading test suite and dependencies packaged in a Hadoop compatible way. This is left to be implemented by the framework using this mode. Additionally these properties may optionally be set if not already in the CLASSPATH; fs.default.name and mapred.job.tracker.

Field Summary

static org.apache.hadoop.hdfs.MiniDFSCluster	dfs
static MiniMRCluster	mr

Fields inherited from class cascading.platform.hadoop.BaseHadoopPlatform

fileSys, jobConf, logger, numMapTasks, numReduceTasks, properties

Fields inherited from class cascading.platform.TestPlatform

CLUSTER_TESTING_PROPERTY

Constructor Summary

HadoopPlatform()

Method Summary

FlowConnector	getFlowConnector(Map<Object,Object> properties)
Integer	getNumMapTasks(Map<Object,Object> properties)
Integer	getNumReduceTasks(Map<Object,Object> properties)
void	setNumMapTasks(Map<Object,Object> properties, int numMapTasks)
void	setNumReduceTasks(Map<Object,Object> properties, int numReduceTasks)
void	setUp()

Methods inherited from class cascading.platform.hadoop.BaseHadoopPlatform

copyFromLocal, copyToLocal, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowProcess, getHiddenTemporaryPath, getJobConf, getLongComparator, getPartitionTap, getProperties, getStringComparator, getTap, getTemplateTap, getTemplateTap, getTestConfigDefScheme, getTestFailScheme, getTextFile, isHDFSAvailable, isMapReduce, remoteExists, remoteRemove, safeFileName, setNumMapTasks, setNumReduceTasks, tearDown

Methods inherited from class cascading.platform.TestPlatform

getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowConnector, getGlobalProperties, getName, getNumMappers, getNumReducers, getTabDelimitedFile, getTabDelimitedFile, getTextFile, getTextFile, getTextFile, getTextFile, isUseCluster, setNumMappers, setNumReducers, setUseCluster

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

dfs

public static transient org.apache.hadoop.hdfs.MiniDFSCluster dfs

mr

public static transient MiniMRCluster mr

Constructor Detail

HadoopPlatform

public HadoopPlatform()

Method Detail

getFlowConnector

public FlowConnector getFlowConnector(Map<Object,Object> properties)

Specified by:

getFlowConnector in class cascading.platform.TestPlatform

setNumMapTasks

public void setNumMapTasks(Map<Object,Object> properties,
                           int numMapTasks)

Overrides:

setNumMapTasks in class cascading.platform.TestPlatform

setNumReduceTasks

public void setNumReduceTasks(Map<Object,Object> properties,
                              int numReduceTasks)

Overrides:

setNumReduceTasks in class cascading.platform.TestPlatform

getNumMapTasks

public Integer getNumMapTasks(Map<Object,Object> properties)

Overrides:

getNumMapTasks in class cascading.platform.TestPlatform

getNumReduceTasks

public Integer getNumReduceTasks(Map<Object,Object> properties)

Overrides:

getNumReduceTasks in class cascading.platform.TestPlatform

setUp

public void setUp()
           throws IOException

Specified by:

setUp in class cascading.platform.TestPlatform

Throws:

IOException