cascading.pipe.assembly
Class Unique

java.lang.Object
  extended by cascading.pipe.Pipe
      extended by cascading.pipe.SubAssembly
          extended by cascading.pipe.assembly.Unique
All Implemented Interfaces:
FlowElement, Serializable

public class Unique
extends SubAssembly

Class Unique SubAssembly is used to filter all duplicates out of a tuple stream.

Typically finding unique value in a tuple stream relies on a GroupBy and a First Aggregator operation.

This SubAssembly also uses the Unique.FilterPartialDuplicates Filter to remove as many observed duplicates before the GroupBy operator to reduce IO over the network.

This strategy is similar to using combiners, except no sorting or serialization is invoked and results in a much simpler mechanism.

The threshold value tells the underlying FilterPartialDuplicates how many values to cache for duplicate comparison before dropping values from the LRU cache.

See Also:
Serialized Form

Nested Class Summary
static class Unique.FilterPartialDuplicates
          Class FilterPartialDuplicates is a Filter that is used to remove observed duplicates from the tuple stream.
 
Field Summary
 
Fields inherited from class cascading.pipe.Pipe
previous
 
Constructor Summary
Unique(Pipe[] pipes, Fields groupingFields)
          Constructor Unique creates a new Unique instance.
Unique(Pipe[] pipes, Fields groupingFields, int threshold)
          Constructor Unique creates a new Unique instance.
Unique(Pipe pipe, Fields groupingFields)
          Constructor Unique creates a new Unique instance.
Unique(Pipe pipe, Fields groupingFields, int threshold)
          Constructor Unique creates a new Unique instance.
Unique(String name, Pipe[] pipes, Fields groupingFields)
          Constructor Unique creates a new Unique instance.
Unique(String name, Pipe[] pipes, Fields groupingFields, int threshold)
          Constructor Unique creates a new Unique instance.
Unique(String name, Pipe pipe, Fields groupingFields)
          Constructor Unique creates a new Unique instance.
Unique(String name, Pipe pipe, Fields groupingFields, int threshold)
          Constructor Unique creates a new Unique instance.
 
Method Summary
 
Methods inherited from class cascading.pipe.SubAssembly
getName, getPrevious, getTailNames, getTails, setTails, unwind
 
Methods inherited from class cascading.pipe.Pipe
equals, getHeads, getTrace, hashCode, isEquivalentTo, names, outgoingScopeFor, pipes, print, printInternal, resolveFields, resolveIncomingOperationFields, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

Unique

@ConstructorProperties(value={"pipe","groupingFields"})
public Unique(Pipe pipe,
                                         Fields groupingFields)
Constructor Unique creates a new Unique instance.

Parameters:
pipe - of type Pipe
groupingFields - of type Fields

Unique

@ConstructorProperties(value={"pipe","groupingFields","threshold"})
public Unique(Pipe pipe,
                                         Fields groupingFields,
                                         int threshold)
Constructor Unique creates a new Unique instance.

Parameters:
pipe - of type Pipe
groupingFields - of type Fields
threshold - of type int

Unique

@ConstructorProperties(value={"name","pipe","groupingFields"})
public Unique(String name,
                                         Pipe pipe,
                                         Fields groupingFields)
Constructor Unique creates a new Unique instance.

Parameters:
name - of type String
pipe - of type Pipe
groupingFields - of type Fields

Unique

@ConstructorProperties(value={"name","pipe","groupingFields","threshold"})
public Unique(String name,
                                         Pipe pipe,
                                         Fields groupingFields,
                                         int threshold)
Constructor Unique creates a new Unique instance.

Parameters:
name - of type String
pipe - of type Pipe
groupingFields - of type Fields
threshold - of type int

Unique

@ConstructorProperties(value={"pipes","groupingFields"})
public Unique(Pipe[] pipes,
                                         Fields groupingFields)
Constructor Unique creates a new Unique instance.

Parameters:
pipes - of type Pipe[]
groupingFields - of type Fields

Unique

@ConstructorProperties(value={"pipes","groupingFields","threshold"})
public Unique(Pipe[] pipes,
                                         Fields groupingFields,
                                         int threshold)
Constructor Unique creates a new Unique instance.

Parameters:
pipes - of type Pipe[]
groupingFields - of type Fields
threshold - of type int

Unique

@ConstructorProperties(value={"name","pipes","groupingFields"})
public Unique(String name,
                                         Pipe[] pipes,
                                         Fields groupingFields)
Constructor Unique creates a new Unique instance.

Parameters:
name - of type String
pipes - of type Pipe[]
groupingFields - of type Fields

Unique

@ConstructorProperties(value={"name","pipes","groupingFields","threshold"})
public Unique(String name,
                                         Pipe[] pipes,
                                         Fields groupingFields,
                                         int threshold)
Constructor Unique creates a new Unique instance.

Parameters:
name - of type String
pipes - of type Pipe[]
groupingFields - of type Fields
threshold - of type int


Copyright © 2007-2010 Concurrent, Inc. All Rights Reserved.