cascading.tuple.collect
Class SpillableTupleMap
java.lang.Object
java.util.AbstractMap<K,V>
java.util.HashMap<Tuple,Collection<Tuple>>
cascading.tuple.collect.SpillableTupleMap
- All Implemented Interfaces:
- Spillable, Serializable, Cloneable, Map<Tuple,Collection<Tuple>>
public abstract class SpillableTupleMap
- extends HashMap<Tuple,Collection<Tuple>>
- implements Spillable
SpillableTupleMap is a HashMap that will allow for multiple values per key, and if the number of values for a given
key reach a specific threshold, they will be spilled to disk using a SpillableTupleList
instance. Only
values are spilled, keys are not spilled and too many keys can result in a OutOfMemoryError
.
The SpillableProps.MAP_THRESHOLD
value sets the number of total values that this map will
strive to keep in memory regardless of the number of keys. This is achieved by dynamically calculating the threshold
used by each child SpillableTupleList instance using
threshold = Min( list_threshold, map_threshold / current_num_keys )
.
To set the list threshold, see SpillableProps
fluent helper class.
This class is used by the HashJoin
pipe, to set properties specific to a given
join instance, see the Pipe.getConfigDef()
method.
- See Also:
HadoopSpillableTupleMap
,
Serialized Form
Constructor Summary |
SpillableTupleMap(int initialCapacity,
float loadFactor,
int mapThreshold,
int initListThreshold)
|
SpillableTupleMap(int mapThreshold,
int initListThreshold)
|
Methods inherited from class java.util.HashMap |
clear, clone, containsKey, containsValue, entrySet, isEmpty, keySet, put, putAll, remove, size, values |
MAP_THRESHOLD
@Deprecated
public static final String MAP_THRESHOLD
- Deprecated.
- The total number of tuple values (not keys) to attempt to keep in memory.
- See Also:
- Constant Field Values
MAP_CAPACITY
@Deprecated
public static final String MAP_CAPACITY
- Deprecated.
- The initial hash map capacity.
- See Also:
HashMap
,
Constant Field Values
MAP_LOADFACTOR
@Deprecated
public static final String MAP_LOADFACTOR
- Deprecated.
- The initial hash map load factor.
- See Also:
HashMap
,
Constant Field Values
SpillableTupleMap
public SpillableTupleMap(int mapThreshold,
int initListThreshold)
SpillableTupleMap
public SpillableTupleMap(int initialCapacity,
float loadFactor,
int mapThreshold,
int initListThreshold)
getMapThreshold
public static int getMapThreshold(FlowProcess flowProcess,
int defaultValue)
getMapCapacity
public static int getMapCapacity(FlowProcess flowProcess,
int defaultValue)
getMapLoadFactor
public static float getMapLoadFactor(FlowProcess flowProcess,
float defaultValue)
getMapThreshold
protected int getMapThreshold()
getInitListThreshold
public int getInitListThreshold()
get
public Collection<Tuple> get(Object object)
- Specified by:
get
in interface Map<Tuple,Collection<Tuple>>
- Overrides:
get
in class HashMap<Tuple,Collection<Tuple>>
createTupleCollection
protected abstract Collection<Tuple> createTupleCollection(Tuple object)
setGrouping
public void setGrouping(Tuple group)
- Specified by:
setGrouping
in interface Spillable
getGrouping
public Tuple getGrouping()
- Specified by:
getGrouping
in interface Spillable
setSpillStrategy
public void setSpillStrategy(Spillable.SpillStrategy spillStrategy)
- Specified by:
setSpillStrategy
in interface Spillable
spillCount
public int spillCount()
- Description copied from interface:
Spillable
- The number of times this container has spilled data to disk.
- Specified by:
spillCount
in interface Spillable
- Returns:
- in int
getSpillListener
public Spillable.SpillListener getSpillListener()
setSpillListener
public void setSpillListener(Spillable.SpillListener spillListener)
- Specified by:
setSpillListener
in interface Spillable
Copyright © 2007-2014 Concurrent, Inc. All Rights Reserved.