cascading.tuple.collect

Class SpillableTupleMap

java.lang.Object

java.util.AbstractMap<K,V>

java.util.HashMap<Tuple,Collection<Tuple>>

cascading.tuple.collect.SpillableTupleMap

All Implemented Interfaces:

Spillable, Serializable, Cloneable, Map<Tuple,Collection<Tuple>>

public abstract class SpillableTupleMap
extends HashMap<Tuple,Collection<Tuple>>
implements Spillable

SpillableTupleMap is a HashMap that will allow for multiple values per key, and if the number of values for a given key reach a specific threshold, they will be spilled to disk using a SpillableTupleList instance. Only values are spilled, keys are not spilled and too many keys can result in a OutOfMemoryError.

The SpillableProps.MAP_THRESHOLD value sets the number of total values that this map will strive to keep in memory regardless of the number of keys. This is achieved by dynamically calculating the threshold used by each child SpillableTupleList instance using threshold = Min( list_threshold, map_threshold / current_num_keys ) .

To set the list threshold, see SpillableProps fluent helper class.

This class is used by the HashJoin pipe, to set properties specific to a given join instance, see the Pipe.getConfigDef() method.

See Also:

HadoopSpillableTupleMap, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class java.util.AbstractMap

AbstractMap.SimpleEntry<K,V>, AbstractMap.SimpleImmutableEntry<K,V>

Nested classes/interfaces inherited from interface cascading.tuple.collect.Spillable

Spillable.SpillListener, Spillable.SpillStrategy

Constructor Summary

Constructors

Constructor and Description
SpillableTupleMap(int initialCapacity, float loadFactor, int mapThreshold, int initListThreshold)
SpillableTupleMap(int mapThreshold, int initListThreshold)

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract Collection<Tuple>	createTupleCollection(Tuple object)
Collection<Tuple>	get(Object object)
Tuple	getGrouping()
int	getInitListThreshold()
static int	getMapCapacity(FlowProcess flowProcess, int defaultValue)
static float	getMapLoadFactor(FlowProcess flowProcess, float defaultValue)
protected int	getMapThreshold()
static int	getMapThreshold(FlowProcess flowProcess, int defaultValue)
Spillable.SpillListener	getSpillListener()
void	setGrouping(Tuple group)
void	setSpillListener(Spillable.SpillListener spillListener)
void	setSpillStrategy(Spillable.SpillStrategy spillStrategy)
int	spillCount() The number of times this container has spilled data to disk.

Methods inherited from class java.util.HashMap

clear, clone, containsKey, containsValue, entrySet, isEmpty, keySet, put, putAll, remove, size, values

Methods inherited from class java.util.AbstractMap

equals, hashCode, toString

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Map

equals, hashCode

Constructor Detail

SpillableTupleMap

public SpillableTupleMap(int mapThreshold,
                 int initListThreshold)

SpillableTupleMap

public SpillableTupleMap(int initialCapacity,
                 float loadFactor,
                 int mapThreshold,
                 int initListThreshold)

Method Detail

getMapThreshold

public static int getMapThreshold(FlowProcess flowProcess,
                  int defaultValue)

getMapCapacity

public static int getMapCapacity(FlowProcess flowProcess,
                 int defaultValue)

getMapLoadFactor

public static float getMapLoadFactor(FlowProcess flowProcess,
                     float defaultValue)

getMapThreshold

protected int getMapThreshold()

getInitListThreshold

public int getInitListThreshold()

get

public Collection<Tuple> get(Object object)

Specified by:

get in interface Map<Tuple,Collection<Tuple>>

Overrides:

get in class HashMap<Tuple,Collection<Tuple>>

createTupleCollection

protected abstract Collection<Tuple> createTupleCollection(Tuple object)

setGrouping

public void setGrouping(Tuple group)

Specified by:

setGrouping in interface Spillable

getGrouping

public Tuple getGrouping()

Specified by:

getGrouping in interface Spillable

setSpillStrategy

public void setSpillStrategy(Spillable.SpillStrategy spillStrategy)

Specified by:

setSpillStrategy in interface Spillable

spillCount

public int spillCount()

Description copied from interface: Spillable

The number of times this container has spilled data to disk.

Specified by:

spillCount in interface Spillable

Returns:

in int

getSpillListener

public Spillable.SpillListener getSpillListener()

setSpillListener

public void setSpillListener(Spillable.SpillListener spillListener)

Specified by:

setSpillListener in interface Spillable