public abstract class SpillableTupleMap extends java.util.HashMap<Tuple,java.util.Collection<Tuple>> implements Spillable
SpillableTupleList
instance. Only
values are spilled, keys are not spilled and too many keys can result in a OutOfMemoryError
.
The SpillableProps.MAP_THRESHOLD
value sets the number of total values that this map will
strive to keep in memory regardless of the number of keys. This is achieved by dynamically calculating the threshold
used by each child SpillableTupleList instance using
threshold = Min( list_threshold, map_threshold / current_num_keys )
.
To set the list threshold, see SpillableProps
fluent helper class.
This class is used by the HashJoin
pipe, to set properties specific to a given
join instance, see the Pipe.getConfigDef()
method.cascading.tuple.hadoop.collect.HadoopSpillableTupleMap
,
Serialized Formjava.util.AbstractMap.SimpleEntry<K,V>, java.util.AbstractMap.SimpleImmutableEntry<K,V>
Spillable.SpillListener, Spillable.SpillStrategy
Constructor and Description |
---|
SpillableTupleMap(int initialCapacity,
float loadFactor,
int mapThreshold,
int initListThreshold) |
SpillableTupleMap(int mapThreshold,
int initListThreshold) |
Modifier and Type | Method and Description |
---|---|
protected abstract java.util.Collection<Tuple> |
createTupleCollection(Tuple object) |
java.util.Collection<Tuple> |
get(java.lang.Object object) |
Tuple |
getGrouping() |
int |
getInitListThreshold() |
static int |
getMapCapacity(FlowProcess flowProcess,
int defaultValue) |
static float |
getMapLoadFactor(FlowProcess flowProcess,
float defaultValue) |
protected int |
getMapThreshold() |
static int |
getMapThreshold(FlowProcess flowProcess,
int defaultValue) |
Spillable.SpillListener |
getSpillListener() |
void |
setGrouping(Tuple group) |
void |
setSpillListener(Spillable.SpillListener spillListener) |
void |
setSpillStrategy(Spillable.SpillStrategy spillStrategy) |
int |
spillCount()
The number of times this container has spilled data to disk.
|
clear, clone, containsKey, containsValue, entrySet, isEmpty, keySet, put, putAll, remove, size, values
public SpillableTupleMap(int mapThreshold, int initListThreshold)
public SpillableTupleMap(int initialCapacity, float loadFactor, int mapThreshold, int initListThreshold)
public static int getMapThreshold(FlowProcess flowProcess, int defaultValue)
public static int getMapCapacity(FlowProcess flowProcess, int defaultValue)
public static float getMapLoadFactor(FlowProcess flowProcess, float defaultValue)
protected int getMapThreshold()
public int getInitListThreshold()
protected abstract java.util.Collection<Tuple> createTupleCollection(Tuple object)
public void setGrouping(Tuple group)
setGrouping
in interface Spillable
public Tuple getGrouping()
getGrouping
in interface Spillable
public void setSpillStrategy(Spillable.SpillStrategy spillStrategy)
setSpillStrategy
in interface Spillable
public int spillCount()
Spillable
spillCount
in interface Spillable
public Spillable.SpillListener getSpillListener()
public void setSpillListener(Spillable.SpillListener spillListener)
setSpillListener
in interface Spillable
Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.