public class Pipe extends Object implements FlowElement, Serializable, Traceable
Each
, Every
, GroupBy
,
CoGroup
, Merge
, HashJoin
, and SubAssembly
.
Pipes are chained together through their constructors.
To effect a split in the pipe,
simply pass a Pipe instance to two or more constructors of subsequent Pipe instances.
A join can be achieved by passing two or more Pipe instances to a CoGroup
or HashJoin
pipe.
A merge can be achieved by passing two or more Pipe instances to a GroupBy
or Merge
pipe.Each
,
Every
,
GroupBy
,
Merge
,
CoGroup
,
HashJoin
,
SubAssembly
,
Serialized FormModifier and Type | Field and Description |
---|---|
protected ConfigDef |
configDef |
protected String |
name
Field name
|
protected ConfigDef |
nodeConfigDef |
protected Pipe |
parent
Field parent
|
protected Pipe |
previous
Field previous
|
protected ConfigDef |
stepConfigDef |
Modifier | Constructor and Description |
---|---|
protected |
Pipe() |
protected |
Pipe(Pipe previous) |
|
Pipe(String name)
Constructor Pipe creates a new Pipe instance with the given name.
|
|
Pipe(String name,
Pipe previous)
Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance.
|
Modifier and Type | Method and Description |
---|---|
boolean |
equals(Object object) |
ConfigDef |
getConfigDef()
Returns a
ConfigDef instance that allows for local properties to be set and made available via
a resulting FlowProcess instance when the pipe is invoked. |
Pipe[] |
getHeads()
Method getHeads returns the first Pipe instances in this pipe assembly.
|
String |
getName()
Get the name of this pipe.
|
ConfigDef |
getNodeConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the pipe is invoked. |
Pipe |
getParent()
Returns the enclosing parent Pipe instance, if any.
|
Pipe[] |
getPrevious()
Get all the upstream pipes this pipe is connected to.
|
ConfigDef |
getStepConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the pipe is invoked. |
String |
getTrace()
Method getTrace returns a String that pinpoints the caller that created this instance.
|
boolean |
hasConfigDef()
Returns
true if there are properties in the configDef instance. |
int |
hashCode() |
boolean |
hasNodeConfigDef()
Returns
true if there are properties in the nodeConfigDef instance. |
boolean |
hasStepConfigDef()
Returns
true if there are properties in the stepConfigDef instance. |
static String |
id(Pipe pipe) |
boolean |
isEquivalentTo(FlowElement element) |
static Pipe[] |
named(String name,
Pipe... tails) |
static String[] |
names(Pipe... tails)
Convenience method for finding all Pipe names in an assembly.
|
Scope |
outgoingScopeFor(Set<Scope> incomingScopes)
Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
|
static Pipe[] |
pipes(Pipe... pipes)
Convenience method to create an array of Pipe instances.
|
String |
print(Scope scope)
Method print is used internally.
|
protected void |
printInternal(StringBuffer buffer,
Scope scope) |
Fields |
resolveIncomingOperationArgumentFields(Scope incomingScope)
Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing Operation arguments.
|
Fields |
resolveIncomingOperationPassThroughFields(Scope incomingScope)
Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing the Pipe outgoing tuple.
|
protected void |
setParent(Pipe parent) |
String |
toString() |
protected ConfigDef stepConfigDef
protected ConfigDef nodeConfigDef
protected Pipe()
@ConstructorProperties(value="previous") protected Pipe(Pipe previous)
@ConstructorProperties(value="name") public Pipe(String name)
name
- name for this branch of Pipes@ConstructorProperties(value={"name","previous"}) public Pipe(String name, Pipe previous)
name
- name for this branch of Pipesprevious
- previous Pipe to receive input Tuples frompublic static Pipe[] pipes(Pipe... pipes)
pipes
- vararg list of pipespublic static String[] names(Pipe... tails)
tails
- vararg list of all tails in given assemblypublic String getName()
public Pipe[] getPrevious()
public Pipe getParent()
SubAssembly
that wraps
this instance.public ConfigDef getConfigDef()
ConfigDef
instance that allows for local properties to be set and made available via
a resulting FlowProcess
instance when the pipe is invoked.
Any properties set on the configDef will not show up in any Flow
or
FlowStep
process level configuration, but will override any of those values as seen by the
current Pipe instance.getConfigDef
in interface FlowElement
public boolean hasConfigDef()
true
if there are properties in the configDef instance.hasConfigDef
in interface FlowElement
public ConfigDef getNodeConfigDef()
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the pipe is invoked.
Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowNode
(in Apache Tez the Vertex configuration). Any value set in the
nodeConfigDef will be overridden by the pipe local #getConfigDef
instance.
Use this method to tweak properties in the process node this pipe instance is planned into. In the case of the
Apache Tez platform, when set on a GroupBy
instance, the number of gather partitions can be modified.
In the case of any Pipe that spans FlowNode boundaries, like GroupBy and CoGroup may on some platforms,
any ConfigDef properties will be applied to the downstream FlowNode. That is, if a GroupBy is the source
to a node, any node ConfigDef properties will be applied. If the GroupBy encountered when applying properties
is on the sink side of a node, the properties will be ignored.getNodeConfigDef
in interface FlowElement
public boolean hasNodeConfigDef()
true
if there are properties in the nodeConfigDef instance.hasNodeConfigDef
in interface FlowElement
public ConfigDef getStepConfigDef()
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the pipe is invoked.
Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowStep
(in Hadoop the MapReduce jobconf). Any value set in the
stepConfigDef will be overridden by the pipe local #getConfigDef
instance.
Use this method to tweak properties in the process step this pipe instance is planned into. In the case of the
Hadoop platform, when set on a GroupBy
instance, the number of reducers can be modified.getStepConfigDef
in interface FlowElement
public boolean hasStepConfigDef()
true
if there are properties in the stepConfigDef instance.hasStepConfigDef
in interface FlowElement
public Pipe[] getHeads()
public Scope outgoingScopeFor(Set<Scope> incomingScopes)
FlowElement
outgoingScopeFor
in interface FlowElement
incomingScopes
- of type Setpublic Fields resolveIncomingOperationArgumentFields(Scope incomingScope)
FlowElement
resolveIncomingOperationArgumentFields
in interface FlowElement
incomingScope
- of type Scopepublic Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)
FlowElement
resolveIncomingOperationPassThroughFields
in interface FlowElement
incomingScope
- of type Scopepublic String getTrace()
Traceable
public boolean isEquivalentTo(FlowElement element)
isEquivalentTo
in interface FlowElement
public String print(Scope scope)
scope
- of type Scopeprotected void printInternal(StringBuffer buffer, Scope scope)
Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.