cascading.pipe
Class Pipe

java.lang.Object
  cascading.pipe.Pipe

All Implemented Interfaces:

FlowElement, Serializable

Direct Known Subclasses:

Checkpoint, Operator, Splice, SubAssembly

public class Pipe
extends Object
implements FlowElement, Serializable

Class Pipe is used to name branches in pipe assemblies, and as a base class for core processing model types, specifically Each, Every, GroupBy, CoGroup, Merge, HashJoin, and SubAssembly.

Pipes are chained together through their constructors.

To effect a split in the pipe, simply pass a Pipe instance to two or more constructors of subsequent Pipe instances.

A join can be achieved by passing two or more Pipe instances to a CoGroup or HashJoin pipe.

A merge can be achieved by passing two or more Pipe instances to a GroupBy or Merge pipe.

See Also:

Each, Every, GroupBy, Merge, CoGroup, HashJoin, SubAssembly, Serialized Form

Field Summary

protected ConfigDef	configDef
protected Pipe	previous Field previous
protected ConfigDef	stepConfigDef

Constructor Summary

protected	Pipe()
protected	Pipe(Pipe previous)
	Pipe(String name) Constructor Pipe creates a new Pipe instance with the given name.
	Pipe(String name, Pipe previous) Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance.

Method Summary

boolean	equals(Object object)
ConfigDef	getConfigDef() Returns a ConfigDef instance that allows for local properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.
Pipe[]	getHeads() Method getHeads returns the first Pipe instances in this pipe assembly.
String	getName() Get the name of this pipe.
Pipe[]	getPrevious() Get all the upstream pipes this pipe is connected to.
ConfigDef	getStepConfigDef() Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.
String	getTrace() Method getTrace returns a String that pinpoint where this instance was created for debugging.
boolean	hasConfigDef() Returns true if there are properties in the configDef instance.
int	hashCode()
boolean	hasStepConfigDef() Returns true if there are properties in the processConfigDef instance.
boolean	isEquivalentTo(FlowElement element)
static Pipe[]	named(String name, Pipe... tails)
static String[]	names(Pipe... tails) Convenience method for finding all Pipe names in an assembly.
cascading.flow.planner.Scope	outgoingScopeFor(Set<cascading.flow.planner.Scope> incomingScopes) Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
static Pipe[]	pipes(Pipe... pipes) Convenience method to create an array of Pipe instances.
String	print(cascading.flow.planner.Scope scope) Method print is used internally.
protected void	printInternal(StringBuffer buffer, cascading.flow.planner.Scope scope)
Fields	resolveIncomingOperationArgumentFields(cascading.flow.planner.Scope incomingScope) Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments.
Fields	resolveIncomingOperationPassThroughFields(cascading.flow.planner.Scope incomingScope) Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

previous

protected Pipe previous

Field previous

configDef

protected ConfigDef configDef

stepConfigDef

protected ConfigDef stepConfigDef

Constructor Detail

Pipe

protected Pipe()

Pipe

@ConstructorProperties(value="previous")
protected Pipe(Pipe previous)

Pipe

@ConstructorProperties(value="name")
public Pipe(String name)

Constructor Pipe creates a new Pipe instance with the given name. This is useful as the 'start' or head of a pipe assembly.

Parameters:

name - name for this branch of Pipes

Pipe

@ConstructorProperties(value={"name","previous"})
public Pipe(String name,
                                       Pipe previous)

Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance. This is useful for naming a branch in a pipe assembly. Or renaming the branch mid-way down.

Parameters:

name - name for this branch of Pipes

previous - previous Pipe to receive input Tuples from

Method Detail

pipes

public static Pipe[] pipes(Pipe... pipes)

Convenience method to create an array of Pipe instances.

Parameters:

pipes - vararg list of pipes

Returns:

array of pipes

names

public static String[] names(Pipe... tails)

Convenience method for finding all Pipe names in an assembly.

Parameters:

tails - vararg list of all tails in given assembly

Returns:

array of Pipe names

named

public static Pipe[] named(String name,
                           Pipe... tails)

getName

public String getName()

Get the name of this pipe. Guaranteed non-null.

Returns:

String the name of this pipe

getPrevious

public Pipe[] getPrevious()

Get all the upstream pipes this pipe is connected to. This method will return the Pipe instances passed on the constructors as inputs to this Pipe instance.

Returns:

all the upstream pipes this pipe is connected to.

getConfigDef

public ConfigDef getConfigDef()

Returns a ConfigDef instance that allows for local properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.

Any properties set on the configDef will not show up in any Flow or FlowStep process level configuration, but will override any of those values as seen by the current Pipe instance.

Specified by:

getConfigDef in interface FlowElement

Returns:

an instance of ConfigDef

hasConfigDef

public boolean hasConfigDef()

Returns true if there are properties in the configDef instance.

Specified by:

hasConfigDef in interface FlowElement

Returns:

true if there are configDef properties

getStepConfigDef

public ConfigDef getStepConfigDef()

Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.

Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in the current process FlowStep (in Hadoop the MapReduce jobconf). Any value set in the stepConfigDef will be overridden by the pipe local #getConfigDef instance.

Use this method to tweak properties in the process step this pipe instance is planned into. In the case of the Hadoop platform, when set on a GroupBy instance, the number of reducers can be modified.

Specified by:

getStepConfigDef in interface FlowElement

Returns:

an instance of ConfigDef

hasStepConfigDef

public boolean hasStepConfigDef()

Returns true if there are properties in the processConfigDef instance.

Specified by:

hasStepConfigDef in interface FlowElement

Returns:

true if there are processConfigDef properties

getHeads

public Pipe[] getHeads()

Method getHeads returns the first Pipe instances in this pipe assembly.

Returns:

the first (type Pipe[]) of this Pipe object.

outgoingScopeFor

public cascading.flow.planner.Scope outgoingScopeFor(Set<cascading.flow.planner.Scope> incomingScopes)

Description copied from interface: FlowElement

Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.

Specified by:

outgoingScopeFor in interface FlowElement

Parameters:

incomingScopes - of type Set

Returns:

Scope

resolveIncomingOperationArgumentFields

public Fields resolveIncomingOperationArgumentFields(cascading.flow.planner.Scope incomingScope)

Description copied from interface: FlowElement

Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments.

Specified by:

resolveIncomingOperationArgumentFields in interface FlowElement

Parameters:

incomingScope - of type Scope

Returns:

Fields

resolveIncomingOperationPassThroughFields

public Fields resolveIncomingOperationPassThroughFields(cascading.flow.planner.Scope incomingScope)

Description copied from interface: FlowElement

Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple.

Specified by:

resolveIncomingOperationPassThroughFields in interface FlowElement

Parameters:

incomingScope - of type Scope

Returns:

Fields

getTrace

public String getTrace()

Method getTrace returns a String that pinpoint where this instance was created for debugging.

Returns:

String

toString

public String toString()

Overrides:

toString in class Object

isEquivalentTo

public boolean isEquivalentTo(FlowElement element)

Specified by:

isEquivalentTo in interface FlowElement

equals

public boolean equals(Object object)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

print

public String print(cascading.flow.planner.Scope scope)

Method print is used internally.

Parameters:

scope - of type Scope

Returns:

String

printInternal

protected void printInternal(StringBuffer buffer,
                             cascading.flow.planner.Scope scope)