cascading.tap.hadoop
Class HfsProps

java.lang.Object
  extended by cascading.property.Props
      extended by cascading.tap.hadoop.HfsProps

public class HfsProps
extends Props

Class HfsProps is a fluent helper for setting various Hadoop FS level properties that some Flow may or may not be required to have set. These properties are typically passed to a Flow via a FlowConnector.


Field Summary
static String COMBINE_INPUT_FILES
          Field COMBINE_INPUT_FILES
static String COMBINE_INPUT_FILES_SAFE_MODE
          Field COMBINE_INPUT_FILES_SAFEMODE
static String COMBINE_INPUT_FILES_SIZE_MAX
          Field COMBINE_INPUT_FILES_SIZE_MAX
protected  Long combinedInputMaxSize
           
protected  Boolean combinedInputSafeMode
           
static String LOCAL_MODE_SCHEME
          Fields LOCAL_MODE_SCHEME *
protected  String localModeScheme
           
static String TEMPORARY_DIRECTORY
          Field TEMPORARY_DIRECTORY
protected  String temporaryDirectory
           
protected  Boolean useCombinedInput
           
 
Constructor Summary
HfsProps()
           
 
Method Summary
protected  void addPropertiesTo(Properties properties)
           
 Long getCombinedInputMaxSize()
           
 String getLocalModeScheme()
           
 String getTemporaryDirectory()
           
static HfsProps hfsProps()
          Creates a new HfsProps instance.
 boolean isUseCombinedInput()
           
 boolean isUseCombinedInputSafeMode()
           
 HfsProps setCombinedInputMaxSize(long combinedInputMaxSize)
          Method setCombinedInputMaxSize sets the maximum input split size to be used.
static void setCombinedInputMaxSize(Map<Object,Object> properties, Long size)
          Method setCombinedInputMaxSize sets the maximum input split size to be used.
static void setLocalModeScheme(Map<Object,Object> properties, String scheme)
          Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode.
 HfsProps setLocalModeScheme(String localModeScheme)
          Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode.
static void setTemporaryDirectory(Map<Object,Object> properties, String temporaryDirectory)
          Method setTemporaryDirectory sets the temporary directory on the given properties object.
 HfsProps setTemporaryDirectory(String temporaryDirectory)
          Method setTemporaryDirectory sets the temporary directory for use on the underlying filesystem.
 HfsProps setUseCombinedInput(boolean useCombinedInput)
          Method setUseCombinedInput provides a means to indicate whether to leverage CombineFileInputFormat for the input format.
static void setUseCombinedInput(Map<Object,Object> properties, Boolean combine)
          Method setUseCombinedInput provides a means to indicate whether to leverage CombineFileInputFormat for the input format.
 HfsProps setUseCombinedInputSafeMode(boolean combinedInputSafeMode)
          Method setUseCombinedInputSafeMode toggles safe mode when using CombineFileInputFormat.
static void setUseCombinedInputSafeMode(Map<Object,Object> properties, Boolean safeMode)
          Method setUseCombinedInputSafeMode toggles safe mode when using CombineFileInputFormat.
 
Methods inherited from class cascading.property.Props
buildProperties, buildProperties, buildProperties, buildProperties, setProperties, setProperties
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

TEMPORARY_DIRECTORY

public static final String TEMPORARY_DIRECTORY
Field TEMPORARY_DIRECTORY

See Also:
Constant Field Values

LOCAL_MODE_SCHEME

public static final String LOCAL_MODE_SCHEME
Fields LOCAL_MODE_SCHEME *

See Also:
Constant Field Values

COMBINE_INPUT_FILES

public static final String COMBINE_INPUT_FILES
Field COMBINE_INPUT_FILES

See Also:
Constant Field Values

COMBINE_INPUT_FILES_SAFE_MODE

public static final String COMBINE_INPUT_FILES_SAFE_MODE
Field COMBINE_INPUT_FILES_SAFEMODE

See Also:
Constant Field Values

COMBINE_INPUT_FILES_SIZE_MAX

public static final String COMBINE_INPUT_FILES_SIZE_MAX
Field COMBINE_INPUT_FILES_SIZE_MAX

See Also:
Constant Field Values

temporaryDirectory

protected String temporaryDirectory

localModeScheme

protected String localModeScheme

useCombinedInput

protected Boolean useCombinedInput

combinedInputMaxSize

protected Long combinedInputMaxSize

combinedInputSafeMode

protected Boolean combinedInputSafeMode
Constructor Detail

HfsProps

public HfsProps()
Method Detail

setTemporaryDirectory

public static void setTemporaryDirectory(Map<Object,Object> properties,
                                         String temporaryDirectory)
Method setTemporaryDirectory sets the temporary directory on the given properties object.

Parameters:
properties - of type Map
temporaryDirectory - of type String

setLocalModeScheme

public static void setLocalModeScheme(Map<Object,Object> properties,
                                      String scheme)
Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode. By default the value is "file", set to "none" to disable entirely.

Parameters:
properties - of type Map
scheme - a String

setUseCombinedInput

public static void setUseCombinedInput(Map<Object,Object> properties,
                                       Boolean combine)
Method setUseCombinedInput provides a means to indicate whether to leverage CombineFileInputFormat for the input format. By default it is false.

Use setCombinedInputMaxSize(long) to set the max split/combined input size. Other specific properties must be specified directly if needed. Specifically "mapred.min.split.size.per.node" and "mapred.min.split.size.per.rack", which are 0 by default.

Parameters:
properties - of type Map
combine - a boolean

setUseCombinedInputSafeMode

public static void setUseCombinedInputSafeMode(Map<Object,Object> properties,
                                               Boolean safeMode)
Method setUseCombinedInputSafeMode toggles safe mode when using CombineFileInputFormat. Safe mode will throw an exception if the underlying InputFormat is not of type FileInputFormat. If safeMode is off a warning will be logged instead. safeMode is on by default.

Setting this property when not setting setUseCombinedInput(boolean) to true has no effect.

Parameters:
properties - of type Map
safeMode - a boolean

setCombinedInputMaxSize

public static void setCombinedInputMaxSize(Map<Object,Object> properties,
                                           Long size)
Method setCombinedInputMaxSize sets the maximum input split size to be used.

This property is an alias for the Hadoop property "mapred.max.split.size".

Parameters:
properties - of type Map
size - of type long

hfsProps

public static HfsProps hfsProps()
Creates a new HfsProps instance.

Returns:
HfsProps instance

getTemporaryDirectory

public String getTemporaryDirectory()

setTemporaryDirectory

public HfsProps setTemporaryDirectory(String temporaryDirectory)
Method setTemporaryDirectory sets the temporary directory for use on the underlying filesystem.

Parameters:
temporaryDirectory - of type String
Returns:
returns this instance

getLocalModeScheme

public String getLocalModeScheme()

setLocalModeScheme

public HfsProps setLocalModeScheme(String localModeScheme)
Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode. By default the value is "file", set to "none" to disable entirely.

Parameters:
localModeScheme - of type String
Returns:
returns this instance

isUseCombinedInput

public boolean isUseCombinedInput()

setUseCombinedInput

public HfsProps setUseCombinedInput(boolean useCombinedInput)
Method setUseCombinedInput provides a means to indicate whether to leverage CombineFileInputFormat for the input format. By default it is false.

Parameters:
useCombinedInput - boolean
Returns:
returns this instance

getCombinedInputMaxSize

public Long getCombinedInputMaxSize()

setCombinedInputMaxSize

public HfsProps setCombinedInputMaxSize(long combinedInputMaxSize)
Method setCombinedInputMaxSize sets the maximum input split size to be used.

This value is not honored unless setUseCombinedInput(boolean) is true.

Parameters:
combinedInputMaxSize - of type long
Returns:
returns this instance

isUseCombinedInputSafeMode

public boolean isUseCombinedInputSafeMode()

setUseCombinedInputSafeMode

public HfsProps setUseCombinedInputSafeMode(boolean combinedInputSafeMode)
Method setUseCombinedInputSafeMode toggles safe mode when using CombineFileInputFormat. Safe mode will throw an exception if the underlying InputFormat is not of type FileInputFormat. If safeMode is off a warning will be logged instead. safeMode is on by default.

Setting this property when not setting setUseCombinedInput(boolean) to true has no effect.

Parameters:
combinedInputSafeMode - boolean
Returns:
returns this instance

addPropertiesTo

protected void addPropertiesTo(Properties properties)
Specified by:
addPropertiesTo in class Props


Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.