|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.pipe.Pipe cascading.pipe.SubAssembly cascading.pipe.assembly.Unique
public class Unique
Class Unique SubAssembly
is used to filter all duplicates out of a tuple stream.
GroupBy
and a First
Aggregator
operation.
This SubAssembly also uses the Unique.FilterPartialDuplicates
Filter
to remove as many observed duplicates before the GroupBy operator to reduce IO over the network.
This strategy is similar to using combiners
, except no sorting or serialization is invoked and results
in a much simpler mechanism.
The threshold
value tells the underlying FilterPartialDuplicates how many values to cache for duplicate
comparison before dropping values from the LRU cache.
Nested Class Summary | |
---|---|
static class |
Unique.FilterPartialDuplicates
Class FilterPartialDuplicates is a Filter that is used to remove observed duplicates from the tuple stream. |
Field Summary |
---|
Fields inherited from class cascading.pipe.Pipe |
---|
previous |
Constructor Summary | |
---|---|
Unique(Pipe[] pipes,
Fields groupingFields)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe[] pipes,
Fields groupingFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields groupingFields)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields groupingFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields groupingFields)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields groupingFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields groupingFields)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields groupingFields,
int threshold)
Constructor Unique creates a new Unique instance. |
Method Summary |
---|
Methods inherited from class cascading.pipe.SubAssembly |
---|
getName, getPrevious, getTailNames, getTails, setTails, unwind |
Methods inherited from class cascading.pipe.Pipe |
---|
equals, getHeads, getTrace, hashCode, isEquivalentTo, names, outgoingScopeFor, pipes, print, printInternal, resolveFields, resolveIncomingOperationFields, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Constructor Detail |
---|
@ConstructorProperties(value={"pipe","groupingFields"}) public Unique(Pipe pipe, Fields groupingFields)
pipe
- of type PipegroupingFields
- of type Fields@ConstructorProperties(value={"pipe","groupingFields","threshold"}) public Unique(Pipe pipe, Fields groupingFields, int threshold)
pipe
- of type PipegroupingFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"name","pipe","groupingFields"}) public Unique(String name, Pipe pipe, Fields groupingFields)
name
- of type Stringpipe
- of type PipegroupingFields
- of type Fields@ConstructorProperties(value={"name","pipe","groupingFields","threshold"}) public Unique(String name, Pipe pipe, Fields groupingFields, int threshold)
name
- of type Stringpipe
- of type PipegroupingFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"pipes","groupingFields"}) public Unique(Pipe[] pipes, Fields groupingFields)
pipes
- of type Pipe[]groupingFields
- of type Fields@ConstructorProperties(value={"pipes","groupingFields","threshold"}) public Unique(Pipe[] pipes, Fields groupingFields, int threshold)
pipes
- of type Pipe[]groupingFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"name","pipes","groupingFields"}) public Unique(String name, Pipe[] pipes, Fields groupingFields)
name
- of type Stringpipes
- of type Pipe[]groupingFields
- of type Fields@ConstructorProperties(value={"name","pipes","groupingFields","threshold"}) public Unique(String name, Pipe[] pipes, Fields groupingFields, int threshold)
name
- of type Stringpipes
- of type Pipe[]groupingFields
- of type Fieldsthreshold
- of type int
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |