|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.pipe.Pipe cascading.pipe.SubAssembly cascading.pipe.assembly.AggregateBy
public class AggregateBy
Class AggregateBy is a SubAssembly
that serves two roles for handling aggregate operations.
SumBy
and CountBy
AggregateBy sub-classes can be performed
in parallel on the same grouping keys.
Custom AggregateBy classes can be created by sub-classing this class and implementing a special
AggregateBy.Functor
for use on the Map side. Multiple Functor instances are managed by the AggregateBy.CompositeFunction
class allowing them all to share the same LRU value map for more efficiency.
To tune the LRU, set the threshold
value to a high enough value to utilize available memory. Or set a
default value via the AGGREGATE_BY_THRESHOLD
property. The current default (AggregateBy.CompositeFunction.DEFAULT_THRESHOLD
)
is 10, 000
unique keys. Note "flushes" from the LRU will be logged in threshold increments along with memory
information.
Note using a AggregateBy instance automatically inserts a GroupBy
into the resulting Flow
.
And passing multiple AggregateBy instances to a parent AggregateBy instance still results in one GroupBy.
Also note that Unique
is not a CompositeAggregator as it makes no sense to combine it with other aggregators,
and so is slightly more optimized internally.
SumBy
,
CountBy
,
Unique
,
Serialized FormNested Class Summary | |
---|---|
static class |
AggregateBy.CompositeFunction
Class CompositeFunction takes multiple Functor instances and manages them as a single Function . |
static class |
AggregateBy.Flush
|
static interface |
AggregateBy.Functor
Interface Functor provides a means to create a simple function for use with the AggregateBy.CompositeFunction class. |
Field Summary | |
---|---|
static String |
AGGREGATE_BY_THRESHOLD
|
Fields inherited from class cascading.pipe.Pipe |
---|
configDef, previous, stepConfigDef |
Constructor Summary | |
---|---|
protected |
AggregateBy(Fields argumentFields,
AggregateBy.Functor functor,
Aggregator aggregator)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
|
AggregateBy(Pipe pipe,
Fields groupingFields,
AggregateBy... assemblies)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
|
AggregateBy(Pipe pipe,
Fields groupingFields,
int threshold,
AggregateBy... assemblies)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
protected |
AggregateBy(String name,
int threshold)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
|
AggregateBy(String name,
Pipe[] pipes,
Fields groupingFields,
AggregateBy... assemblies)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
protected |
AggregateBy(String name,
Pipe[] pipes,
Fields groupingFields,
Fields argumentFields,
AggregateBy.Functor functor,
Aggregator aggregator,
int threshold)
|
|
AggregateBy(String name,
Pipe[] pipes,
Fields groupingFields,
int threshold,
AggregateBy... assemblies)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
|
AggregateBy(String name,
Pipe pipe,
Fields groupingFields,
int threshold,
AggregateBy... assemblies)
Constructor CompositeAggregator creates a new CompositeAggregator instance. |
Method Summary | |
---|---|
protected Aggregator[] |
getAggregators()
|
protected Fields[] |
getArgumentFields()
|
protected AggregateBy.Functor[] |
getFunctors()
|
GroupBy |
getGroupBy()
Method getGroupBy returns the internal GroupBy instance so that any custom properties
can be set on it via Pipe.getStepConfigDef() . |
protected void |
initialize(Fields groupingFields,
Pipe[] pipes,
Fields[] argumentFields,
AggregateBy.Functor[] functors,
Aggregator[] aggregators)
|
protected void |
initialize(Fields groupingFields,
Pipe[] pipes,
Fields argumentFields,
AggregateBy.Functor functor,
Aggregator aggregator)
|
protected void |
verify()
Method verify should be overridden by sub-classes if any values must be tested before the calling constructor returns. |
Methods inherited from class cascading.pipe.SubAssembly |
---|
getName, getPrevious, getTailNames, getTails, setTails, unwind |
Methods inherited from class cascading.pipe.Pipe |
---|
equals, getConfigDef, getHeads, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveFields, resolveIncomingOperationFields, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
public static final String AGGREGATE_BY_THRESHOLD
Constructor Detail |
---|
protected AggregateBy(String name, int threshold)
name
- of type Stringthreshold
- of type intprotected AggregateBy(Fields argumentFields, AggregateBy.Functor functor, Aggregator aggregator)
argumentFields
- of type Fieldsfunctor
- of type Functoraggregator
- of type Aggregatorpublic AggregateBy(Pipe pipe, Fields groupingFields, AggregateBy... assemblies)
pipe
- of type PipegroupingFields
- of type Fieldsassemblies
- of type CompositeAggregator...public AggregateBy(Pipe pipe, Fields groupingFields, int threshold, AggregateBy... assemblies)
pipe
- of type PipegroupingFields
- of type Fieldsthreshold
- of type intassemblies
- of type CompositeAggregator...public AggregateBy(String name, Pipe pipe, Fields groupingFields, int threshold, AggregateBy... assemblies)
pipe
- of type PipegroupingFields
- of type Fieldsthreshold
- of type intassemblies
- of type CompositeAggregator...public AggregateBy(String name, Pipe[] pipes, Fields groupingFields, AggregateBy... assemblies)
name
- of type Stringpipes
- of type Pipe[]groupingFields
- of type Fieldsassemblies
- of type CompositeAggregator...public AggregateBy(String name, Pipe[] pipes, Fields groupingFields, int threshold, AggregateBy... assemblies)
name
- of type Stringpipes
- of type Pipe[]groupingFields
- of type Fieldsthreshold
- of type intassemblies
- of type CompositeAggregator...protected AggregateBy(String name, Pipe[] pipes, Fields groupingFields, Fields argumentFields, AggregateBy.Functor functor, Aggregator aggregator, int threshold)
Method Detail |
---|
protected void initialize(Fields groupingFields, Pipe[] pipes, Fields argumentFields, AggregateBy.Functor functor, Aggregator aggregator)
protected void initialize(Fields groupingFields, Pipe[] pipes, Fields[] argumentFields, AggregateBy.Functor[] functors, Aggregator[] aggregators)
protected void verify()
protected Fields[] getArgumentFields()
protected AggregateBy.Functor[] getFunctors()
protected Aggregator[] getAggregators()
public GroupBy getGroupBy()
GroupBy
instance so that any custom properties
can be set on it via Pipe.getStepConfigDef()
.
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |