|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.pipe.Pipe cascading.pipe.SubAssembly cascading.pipe.assembly.Unique
public class Unique
Class Unique SubAssembly
is used to filter all duplicates out of a tuple stream.
GroupBy
and a FirstNBuffer
Buffer
operation.
If the include
value is set to Unique.Include.NO_NULLS
, any tuple consisting of only null
values will be removed from the stream.
This SubAssembly uses the Unique.FilterPartialDuplicates
Filter
to remove as many observed duplicates before the GroupBy operator to reduce IO over the network.
This strategy is similar to using combiners
, except no sorting or serialization is invoked and results
in a much simpler mechanism.
The threshold
value tells the underlying FilterPartialDuplicates how many values to cache for duplicate
comparison before dropping values from the LRU cache.
Nested Class Summary | |
---|---|
static class |
Unique.FilterPartialDuplicates
Class FilterPartialDuplicates is a Filter that is used to remove observed duplicates from the tuple stream. |
static class |
Unique.Include
|
Field Summary |
---|
Fields inherited from class cascading.pipe.Pipe |
---|
configDef, name, parent, stepConfigDef |
Constructor Summary | |
---|---|
Unique(Pipe[] pipes,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe[] pipes,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe[] pipes,
Fields uniqueFields,
Unique.Include include)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe[] pipes,
Fields uniqueFields,
Unique.Include include,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields uniqueFields,
Unique.Include include)
Constructor Unique creates a new Unique instance. |
|
Unique(Pipe pipe,
Fields uniqueFields,
Unique.Include include,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields uniqueFields,
Unique.Include include)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe[] pipes,
Fields uniqueFields,
Unique.Include include,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields uniqueFields)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields uniqueFields,
int threshold)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields uniqueFields,
Unique.Include include)
Constructor Unique creates a new Unique instance. |
|
Unique(String name,
Pipe pipe,
Fields uniqueFields,
Unique.Include include,
int threshold)
Constructor Unique creates a new Unique instance. |
Method Summary |
---|
Methods inherited from class cascading.pipe.SubAssembly |
---|
getName, getPrevious, getTailNames, getTails, setPrevious, setTails, unwind |
Methods inherited from class cascading.pipe.Pipe |
---|
equals, getConfigDef, getHeads, getParent, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, id, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Constructor Detail |
---|
@ConstructorProperties(value={"pipe","uniqueFields"}) public Unique(Pipe pipe, Fields uniqueFields)
pipe
- of type PipeuniqueFields
- of type Fields@ConstructorProperties(value={"pipe","uniqueFields","include"}) public Unique(Pipe pipe, Fields uniqueFields, Unique.Include include)
pipe
- of type PipeuniqueFields
- of type Fieldsinclude
- of type Include@ConstructorProperties(value={"pipe","uniqueFields","threshold"}) public Unique(Pipe pipe, Fields uniqueFields, int threshold)
pipe
- of type PipeuniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"pipe","uniqueFields","include","threshold"}) public Unique(Pipe pipe, Fields uniqueFields, Unique.Include include, int threshold)
pipe
- of type PipeuniqueFields
- of type Fieldsinclude
- of type Includethreshold
- of type int@ConstructorProperties(value={"name","pipe","uniqueFields"}) public Unique(String name, Pipe pipe, Fields uniqueFields)
name
- of type Stringpipe
- of type PipeuniqueFields
- of type Fields@ConstructorProperties(value={"name","pipe","uniqueFields","include"}) public Unique(String name, Pipe pipe, Fields uniqueFields, Unique.Include include)
name
- of type Stringpipe
- of type PipeuniqueFields
- of type Fieldsinclude
- of type Include@ConstructorProperties(value={"name","pipe","uniqueFields","threshold"}) public Unique(String name, Pipe pipe, Fields uniqueFields, int threshold)
name
- of type Stringpipe
- of type PipeuniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"name","pipe","uniqueFields","include","threshold"}) public Unique(String name, Pipe pipe, Fields uniqueFields, Unique.Include include, int threshold)
name
- of type Stringpipe
- of type PipeuniqueFields
- of type Fieldsinclude
- of type Includethreshold
- of type int@ConstructorProperties(value={"pipes","uniqueFields"}) public Unique(Pipe[] pipes, Fields uniqueFields)
pipes
- of type Pipe[]uniqueFields
- of type Fields@ConstructorProperties(value={"pipes","uniqueFields","include"}) public Unique(Pipe[] pipes, Fields uniqueFields, Unique.Include include)
pipes
- of type Pipe[]uniqueFields
- of type Fieldsinclude
- of type Include@ConstructorProperties(value={"pipes","uniqueFields","threshold"}) public Unique(Pipe[] pipes, Fields uniqueFields, int threshold)
pipes
- of type Pipe[]uniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"pipes","uniqueFields","threshold"}) public Unique(Pipe[] pipes, Fields uniqueFields, Unique.Include include, int threshold)
pipes
- of type Pipe[]uniqueFields
- of type Fieldsinclude
- of type Includethreshold
- of type int@ConstructorProperties(value={"name","pipes","uniqueFields"}) public Unique(String name, Pipe[] pipes, Fields uniqueFields)
name
- of type Stringpipes
- of type Pipe[]uniqueFields
- of type Fields@ConstructorProperties(value={"name","pipes","uniqueFields","include"}) public Unique(String name, Pipe[] pipes, Fields uniqueFields, Unique.Include include)
name
- of type Stringpipes
- of type Pipe[]uniqueFields
- of type Fieldsinclude
- of type Include@ConstructorProperties(value={"name","pipes","uniqueFields","threshold"}) public Unique(String name, Pipe[] pipes, Fields uniqueFields, int threshold)
name
- of type Stringpipes
- of type Pipe[]uniqueFields
- of type Fieldsthreshold
- of type int@ConstructorProperties(value={"name","pipes","uniqueFields","include","threshold"}) public Unique(String name, Pipe[] pipes, Fields uniqueFields, Unique.Include include, int threshold)
name
- of type Stringpipes
- of type Pipe[]uniqueFields
- of type Fieldsthreshold
- of type int
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |