public class HadoopPlatform extends BaseHadoopPlatform<JobConf>
PlatformTestCase
instance
so that all *PlatformTest classes can be tested against Apache Hadoop.
This platform works in three modes.
Hadoop standalone mode is when Hadoop is NOT run as a cluster, and all
child tasks are in process and in memory of the "client" side code.
Hadoop mini cluster mode where a cluster is created on demand using the Hadoop MiniDFSCluster and MiniMRCluster
utilities. When a PlatformTestCase requests to use a cluster, this is the default cluster. All properties are
pulled from the current CLASSPATH via the JobConf.
Lastly remote cluster mode is enabled when the System property "mapred.jar" is set. This is a Hadoop property
specifying the Hadoop "job jar" to be used cluster side. This MUST be the Cascading test suite and dependencies
packaged in a Hadoop compatible way. This is left to be implemented by the framework using this mode. Additionally
these properties may optionally be set if not already in the CLASSPATH; fs.default.name and mapred.job.tracker.Modifier and Type | Field and Description |
---|---|
static org.apache.hadoop.hdfs.MiniDFSCluster |
dfs |
static MiniMRCluster |
mr |
configuration, fileSys, logger, properties
Constructor and Description |
---|
HadoopPlatform() |
Modifier and Type | Method and Description |
---|---|
JobConf |
getConfiguration() |
cascading.flow.FlowConnector |
getFlowConnector(java.util.Map<java.lang.Object,java.lang.Object> properties) |
cascading.flow.FlowProcess |
getFlowProcess() |
java.lang.Integer |
getNumMapTasks(java.util.Map<java.lang.Object,java.lang.Object> properties) |
java.lang.Integer |
getNumReduceTasks(java.util.Map<java.lang.Object,java.lang.Object> properties) |
void |
setNumMapTasks(java.util.Map<java.lang.Object,java.lang.Object> properties,
int numMapTasks) |
void |
setNumReduceTasks(java.util.Map<java.lang.Object,java.lang.Object> properties,
int numReduceTasks) |
void |
setUp() |
copyFromLocal, copyToLocal, getDelimitedFile, getDelimitedFile, getDelimitedFile, getHiddenTemporaryPath, getLongComparator, getPartitionTap, getProperties, getStringComparator, getTap, getTestConfigDefScheme, getTestFailScheme, getTextFile, isHDFSAvailable, isMapReduce, remoteExists, remoteRemove, safeFileName, setNumGatherPartitions, setNumMappers, setNumReducers, tearDown
getApplicationJar, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowConnector, getGlobalProperties, getName, getNumGatherPartitions, getNumGatherPartitionTasks, getNumMappers, getNumReducers, getTabDelimitedFile, getTabDelimitedFile, getTextFile, getTextFile, getTextFile, getTextFile, isDAG, isUseCluster, setNumGatherPartitionTasks, setUseCluster, supportsGroupByAfterMerge
public static transient org.apache.hadoop.hdfs.MiniDFSCluster dfs
public static transient MiniMRCluster mr
public HadoopPlatform()
public cascading.flow.FlowConnector getFlowConnector(java.util.Map<java.lang.Object,java.lang.Object> properties)
getFlowConnector
in class cascading.platform.TestPlatform
public void setNumMapTasks(java.util.Map<java.lang.Object,java.lang.Object> properties, int numMapTasks)
setNumMapTasks
in class cascading.platform.TestPlatform
public void setNumReduceTasks(java.util.Map<java.lang.Object,java.lang.Object> properties, int numReduceTasks)
setNumReduceTasks
in class cascading.platform.TestPlatform
public java.lang.Integer getNumMapTasks(java.util.Map<java.lang.Object,java.lang.Object> properties)
getNumMapTasks
in class cascading.platform.TestPlatform
public java.lang.Integer getNumReduceTasks(java.util.Map<java.lang.Object,java.lang.Object> properties)
getNumReduceTasks
in class cascading.platform.TestPlatform
public JobConf getConfiguration()
getConfiguration
in class BaseHadoopPlatform<JobConf>
public cascading.flow.FlowProcess getFlowProcess()
getFlowProcess
in class cascading.platform.TestPlatform
public void setUp() throws java.io.IOException
setUp
in class cascading.platform.TestPlatform
java.io.IOException
Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.