|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.pipe.Pipe cascading.pipe.Group cascading.pipe.CoGroup
public class CoGroup
The CoGroup pipe allows for two or more tuple streams to join into a single stream.
For every incomingPipe
instance, a Fields
instance must be specified that denotes the field names
or positions that should be co-grouped with the other given Pipe instances. If the incoming Pipe instances declare
one or more field with the same name, the declaredFields must be given to name the outgoing Tuple stream fields
to overcome field name collisions.
resultGroupFields
value sets the resulting grouping field names. By default, if all 'groupFields' values
are equal, that value will be used for resultGroupFields.
By default CoGroup performs an inner join via the InnerJoin
Joiner
class.
Self joins can be achieved by using a constructor that takes a single Pipe and a numSelfJoins value. A value of
1 for numSelfJoins will join the Pipe with itself once.
The outgoing grouping Tuple stream is sorted by the natural order of the grouping fields. To control this order,
at least the first groupingFields value given should be an instance of Fields
containing
Comparator
instances for the appropriate fields.
This allows fine grained control of the sort grouping order.
CoGrouping, specifically joins, do not scale well when implemented over MapReduce. In Cascading there are two
ways to optimize CoGrouping.
The first is to join streams with more data on the left hand side to streams with more sparse data on the right
hand side. That is, always attempt to effect M x N joins where M is large and N is small, instead of where M is
small and N is large. This typically will allow Cascading to forgo spilling a group via the SpillableTupleList
internally.
If spills are happening, consider increasing the spill threshold (CoGroupClosure.SPILL_THRESHOLD
via the properties handed to FlowConnector
.
InnerJoin
,
OuterJoin
,
LeftJoin
,
RightJoin
,
MixedJoin
,
Fields
,
SpillableTupleList
,
CoGroupClosure
,
Serialized FormField Summary |
---|
Fields inherited from class cascading.pipe.Group |
---|
declaredFields, groupFieldsMap, resultGroupFields, sortFieldsMap |
Fields inherited from class cascading.pipe.Pipe |
---|
previous |
Constructor Summary | |
---|---|
CoGroup(Pipe... pipes)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe[] pipes,
Fields[] groupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe[] pipes,
Fields[] groupFields,
Fields declaredFields,
Fields resultGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe[] pipes,
Fields[] groupFields,
Fields declaredFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe pipe,
Fields groupFields,
int numSelfJoins)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields,
Fields resultGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields,
Fields resultGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(Pipe pipe,
Fields groupFields,
int numSelfJoins,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields,
Fields resultGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields,
Fields resultGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe... pipes)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe[] pipes,
Fields[] groupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe[] pipes,
Fields[] groupFields,
Fields declaredFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe[] pipes,
Fields[] groupFields,
Fields declaredFields,
Fields resultGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe[] pipes,
Fields[] groupFields,
Fields declaredFields,
Fields resultGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe[] pipes,
Fields[] groupFields,
Fields declaredFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe pipe,
Fields groupFields,
int numSelfJoins)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(String groupName,
Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(String groupName,
Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields,
Fields resultGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields,
Fields resultGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe pipe,
Fields groupFields,
int numSelfJoins,
Fields declaredFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(String groupName,
Pipe pipe,
Fields groupFields,
int numSelfJoins,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance that performs numSelfJoins number of self joins on the given Pipe instance. |
|
CoGroup(String groupName,
Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields,
Fields resultGroupFields)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields,
Fields resultGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Fields declaredFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
|
CoGroup(String groupName,
Pipe lhs,
Fields lhsGroupFields,
Pipe rhs,
Fields rhsGroupFields,
Joiner joiner)
Constructor CoGroup creates a new CoGroup instance. |
Method Summary |
---|
Methods inherited from class cascading.pipe.Group |
---|
collectReduceGrouping, equals, getDeclaredFields, getGroupingSelectors, getName, getPrevious, getSortingSelectors, hashCode, initializeReduce, isEquivalentTo, isGroupBy, isSorted, isSortReversed, iterateReduceValues, outgoingScopeFor, printInternal, resolveFields, toString, unwrapGrouping |
Methods inherited from class cascading.pipe.Pipe |
---|
getHeads, getTrace, names, pipes, print, resolveIncomingOperationFields |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Constructor Detail |
---|
@ConstructorProperties(value={"lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields"}) public CoGroup(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields)
lhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type Fields@ConstructorProperties(value={"lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields","resultGroupFields"}) public CoGroup(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Fields resultGroupFields)
lhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type FieldsresultGroupFields
- of type Fields@ConstructorProperties(value={"lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields","joiner"}) public CoGroup(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Joiner joiner)
lhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields","resultGroupFields","joiner"}) public CoGroup(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Fields resultGroupFields, Joiner joiner)
lhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type FieldsresultGroupFields
- of type Fieldsjoiner
- of type Joiner@ConstructorProperties(value={"lhs","lhsGroupFields","rhs","rhsGroupFields","joiner"}) public CoGroup(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Joiner joiner)
lhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"lhs","lhsGroupFields","rhs","rhsGroupFields"}) public CoGroup(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields)
lhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type Fields@ConstructorProperties(value="pipes") public CoGroup(Pipe... pipes)
pipes
- of type Pipe...@ConstructorProperties(value={"pipes","groupFields"}) public CoGroup(Pipe[] pipes, Fields[] groupFields)
pipes
- of type Pipe[]groupFields
- of type Fields[]@ConstructorProperties(value={"pipes","groupFields","declaredFields","joiner"}) public CoGroup(Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Joiner joiner)
pipes
- of type Pipe[]groupFields
- of type Fields[]declaredFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"pipes","groupFields","declaredFields","resultGroupFields","joiner"}) public CoGroup(Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Fields resultGroupFields, Joiner joiner)
pipes
- of type Pipe[]groupFields
- of type Fields[]declaredFields
- of type FieldsresultGroupFields
- of type Fieldsjoiner
- of type Joiner@ConstructorProperties(value={"groupName","pipes","groupFields"}) public CoGroup(String groupName, Pipe[] pipes, Fields[] groupFields)
groupName
- of type Stringpipes
- of type Pipe[]groupFields
- of type Fields[]@ConstructorProperties(value={"groupName","pipes","groupFields","declaredFields"}) public CoGroup(String groupName, Pipe[] pipes, Fields[] groupFields, Fields declaredFields)
groupName
- of type Stringpipes
- of type Pipe[]groupFields
- of type Fields[]declaredFields
- of type Fields@ConstructorProperties(value={"groupName","pipes","groupFields","declaredFields","resultGroupFields"}) public CoGroup(String groupName, Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Fields resultGroupFields)
groupName
- of type Stringpipes
- of type Pipe[]groupFields
- of type Fields[]declaredFields
- of type FieldsresultGroupFields
- of type Fields@ConstructorProperties(value={"groupName","pipes","groupFields","declaredFields","joiner"}) public CoGroup(String groupName, Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Joiner joiner)
groupName
- of type Stringpipes
- of type Pipe[]groupFields
- of type Fields[]declaredFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"groupName","pipes","groupFields","declaredFields","resultGroupFields","joiner"}) public CoGroup(String groupName, Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Fields resultGroupFields, Joiner joiner)
groupName
- of type Stringpipes
- of type Pipe[]groupFields
- of type Fields[]declaredFields
- of type FieldsresultGroupFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"groupName","lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields"}) public CoGroup(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields)
groupName
- of type Stringlhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type Fields@ConstructorProperties(value={"groupName","lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields","resultGroupFields"}) public CoGroup(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Fields resultGroupFields)
groupName
- of type Stringlhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type FieldsresultGroupFields
- of type Fields@ConstructorProperties(value={"groupName","lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields","joiner"}) public CoGroup(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Joiner joiner)
groupName
- of type Stringlhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"groupName","lhs","lhsGroupFields","rhs","rhsGroupFields","declaredFields","resultGroupFields","joiner"}) public CoGroup(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Fields resultGroupFields, Joiner joiner)
groupName
- of type Stringlhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type FieldsdeclaredFields
- of type FieldsresultGroupFields
- of type Fieldsjoiner
- of type Joiner@ConstructorProperties(value={"groupName","lhs","lhsGroupFields","rhs","rhsGroupFields","joiner"}) public CoGroup(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Joiner joiner)
groupName
- of type Stringlhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"groupName","lhs","lhsGroupFields","rhs","rhsGroupFields"}) public CoGroup(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields)
groupName
- of type Stringlhs
- of type PipelhsGroupFields
- of type Fieldsrhs
- of type PiperhsGroupFields
- of type Fields@ConstructorProperties(value={"groupName","pipes"}) public CoGroup(String groupName, Pipe... pipes)
groupName
- of type Stringpipes
- of type Pipe...@ConstructorProperties(value={"pipe","groupFields","numSelfJoins","declaredFields"}) public CoGroup(Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields)
Pipe
instance.
pipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type Fields@ConstructorProperties(value={"pipe","groupFields","numSelfJoins","declaredFields","resultGroupFields"}) public CoGroup(Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Fields resultGroupFields)
pipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type FieldsresultGroupFields
- of type Fields@ConstructorProperties(value={"pipe","groupFields","numSelfJoins","declaredFields","joiner"}) public CoGroup(Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Joiner joiner)
Pipe
instance.
pipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"pipe","groupFields","numSelfJoins","declaredFields","resultGroupFields","joiner"}) public CoGroup(Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Fields resultGroupFields, Joiner joiner)
pipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type FieldsresultGroupFields
- of type Fieldsjoiner
- of type Joiner@ConstructorProperties(value={"pipe","groupFields","numSelfJoins","joiner"}) public CoGroup(Pipe pipe, Fields groupFields, int numSelfJoins, Joiner joiner)
Pipe
instance.
pipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intjoiner
- of type CoGrouper@ConstructorProperties(value={"pipe","groupFields","numSelfJoins"}) public CoGroup(Pipe pipe, Fields groupFields, int numSelfJoins)
Pipe
instance.
pipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type int@ConstructorProperties(value={"groupName","pipe","groupFields","numSelfJoins","declaredFields"}) public CoGroup(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields)
Pipe
instance.
groupName
- of type Stringpipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type Fields@ConstructorProperties(value={"groupName","pipe","groupFields","numSelfJoins","declaredFields","resultGroupFields"}) public CoGroup(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Fields resultGroupFields)
groupName
- of type Stringpipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type FieldsresultGroupFields
- of type Fields@ConstructorProperties(value={"groupName","pipe","groupFields","numSelfJoins","declaredFields","joiner"}) public CoGroup(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Joiner joiner)
Pipe
instance.
groupName
- of type Stringpipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type Fieldsjoiner
- of type CoGrouper@ConstructorProperties(value={"groupName","pipe","groupFields","numSelfJoins","declaredFields","resultGroupFields","joiner"}) public CoGroup(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Fields resultGroupFields, Joiner joiner)
groupName
- of type Stringpipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intdeclaredFields
- of type FieldsresultGroupFields
- of type Fieldsjoiner
- of type Joiner@ConstructorProperties(value={"groupName","pipe","groupFields","numSelfJoins","joiner"}) public CoGroup(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Joiner joiner)
Pipe
instance.
groupName
- of type Stringpipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type intjoiner
- of type CoGrouper@ConstructorProperties(value={"groupName","pipe","groupFields","numSelfJoins"}) public CoGroup(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins)
Pipe
instance.
groupName
- of type Stringpipe
- of type PipegroupFields
- of type FieldsnumSelfJoins
- of type int
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |