|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.pipe.Pipe cascading.pipe.SubAssembly cascading.pipe.assembly.Unique
public class Unique
Class Unique SubAssembly
is used to filter all duplicates out of a tuple stream.
GroupBy
and a First
Aggregator
operation.
This SubAssembly uses the Unique.FilterPartialDuplicates
Filter
to remove as many observed duplicates before the GroupBy operator to reduce IO over the network.
This strategy is similar to using combiners
, except no sorting or serialization is invoked and results
in a much simpler mechanism.
The threshold
value tells the underlying FilterPartialDuplicates how many values to cache for duplicate
comparison before dropping values from the LRU cache.
Nested Class Summary | |
---|---|
static class |
Unique.FilterPartialDuplicates
Class FilterPartialDuplicates is a Filter that is used to remove observed duplicates from the tuple stream. |
Field Summary |
---|
Fields inherited from class cascading.pipe.Pipe |
---|
configDef, previous, stepConfigDef |
Constructor Summary | |
---|---|
Unique(Pipe[] pipes,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe[] pipes,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
Method Summary |
---|
Methods inherited from class cascading.pipe.SubAssembly |
---|
getName, getPrevious, getTailNames, getTails, setTails, unwind |
Methods inherited from class cascading.pipe.Pipe |
---|
equals, getConfigDef, getHeads, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveFields, resolveIncomingOperationFields, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Constructor Detail |
---|
@ConstructorProperties(value={"pipe","uniqueFields"}) public Unique(Pipe pipe, Fields uniqueFields)
pipe
- of type PipeuniqueFields
- of type Fields@ConstructorProperties(value={"pipe","uniqueFields","threshold"}) public Unique(Pipe pipe, Fields uniqueFields, int threshold)
pipe
- of type PipeuniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"name","pipe","uniqueFields"}) public Unique(String name, Pipe pipe, Fields uniqueFields)
name
- of type Stringpipe
- of type PipeuniqueFields
- of type Fields@ConstructorProperties(value={"name","pipe","uniqueFields","threshold"}) public Unique(String name, Pipe pipe, Fields uniqueFields, int threshold)
name
- of type Stringpipe
- of type PipeuniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"pipes","uniqueFields"}) public Unique(Pipe[] pipes, Fields uniqueFields)
pipes
- of type Pipe[]uniqueFields
- of type Fields@ConstructorProperties(value={"pipes","uniqueFields","threshold"}) public Unique(Pipe[] pipes, Fields uniqueFields, int threshold)
pipes
- of type Pipe[]uniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"name","pipes","uniqueFields"}) public Unique(String name, Pipe[] pipes, Fields uniqueFields)
name
- of type Stringpipes
- of type Pipe[]uniqueFields
- of type Fields@ConstructorProperties(value={"name","pipes","uniqueFields","threshold"}) public Unique(String name, Pipe[] pipes, Fields uniqueFields, int threshold)
name
- of type Stringpipes
- of type Pipe[]uniqueFields
- of type Fieldsthreshold
- of type int
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |