Class HadoopPlatform

  extended by cascading.platform.TestPlatform
      extended by cascading.platform.hadoop.BaseHadoopPlatform
          extended by cascading.platform.hadoop.HadoopPlatform

public class HadoopPlatform
extends BaseHadoopPlatform

Class HadoopPlatform is automatically loaded and injected into a PlatformTestCase instance so that all *PlatformTest classes can be tested against Apache Hadoop.

This platform works in three modes.

Hadoop standalone mode is when Hadoop is NOT run as a cluster, and all child tasks are in process and in memory of the "client" side code.

Hadoop mini cluster mode where a cluster is created on demand using the Hadoop MiniDFSCluster and MiniMRCluster utilities. When a PlatformTestCase requests to use a cluster, this is the default cluster. All properties are pulled from the current CLASSPATH via the JobConf.

Lastly remote cluster mode is enabled when the System property "mapred.jar" is set. This is a Hadoop property specifying the Hadoop "job jar" to be used cluster side. This MUST be the Cascading test suite and dependencies packaged in a Hadoop compatible way. This is left to be implemented by the framework using this mode. Additionally these properties may optionally be set if not already in the CLASSPATH; and mapred.job.tracker.

Field Summary
static org.apache.hadoop.hdfs.MiniDFSCluster dfs
static MiniMRCluster mr
Fields inherited from class cascading.platform.hadoop.BaseHadoopPlatform
fileSys, jobConf, logger, numMapTasks, numReduceTasks, properties
Fields inherited from class cascading.platform.TestPlatform
Constructor Summary
Method Summary
 FlowConnector getFlowConnector(Map<Object,Object> properties)
 Integer getNumMapTasks(Map<Object,Object> properties)
 Integer getNumReduceTasks(Map<Object,Object> properties)
 void setNumMapTasks(Map<Object,Object> properties, int numMapTasks)
 void setNumReduceTasks(Map<Object,Object> properties, int numReduceTasks)
 void setUp()
Methods inherited from class cascading.platform.hadoop.BaseHadoopPlatform
copyFromLocal, copyToLocal, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowProcess, getHiddenTemporaryPath, getJobConf, getLongComparator, getPartitionTap, getProperties, getStringComparator, getTap, getTemplateTap, getTemplateTap, getTestConfigDefScheme, getTestFailScheme, getTextFile, isHDFSAvailable, isMapReduce, remoteExists, remoteRemove, safeFileName, setNumMapTasks, setNumReduceTasks, tearDown
Methods inherited from class cascading.platform.TestPlatform
getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowConnector, getGlobalProperties, getName, getNumMappers, getNumReducers, getTabDelimitedFile, getTabDelimitedFile, getTextFile, getTextFile, getTextFile, getTextFile, isUseCluster, setNumMappers, setNumReducers, setUseCluster
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail


public static transient org.apache.hadoop.hdfs.MiniDFSCluster dfs


public static transient MiniMRCluster mr
Constructor Detail


public HadoopPlatform()
Method Detail


public FlowConnector getFlowConnector(Map<Object,Object> properties)
Specified by:
getFlowConnector in class cascading.platform.TestPlatform


public void setNumMapTasks(Map<Object,Object> properties,
                           int numMapTasks)
setNumMapTasks in class cascading.platform.TestPlatform


public void setNumReduceTasks(Map<Object,Object> properties,
                              int numReduceTasks)
setNumReduceTasks in class cascading.platform.TestPlatform


public Integer getNumMapTasks(Map<Object,Object> properties)
getNumMapTasks in class cascading.platform.TestPlatform


public Integer getNumReduceTasks(Map<Object,Object> properties)
getNumReduceTasks in class cascading.platform.TestPlatform


public void setUp()
           throws IOException
Specified by:
setUp in class cascading.platform.TestPlatform

Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.