|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object cascading.tap.Tap<Config,Input,Output> cascading.tap.DecoratorTap<MetaInfo,Config,Input,Output>
public class DecoratorTap<MetaInfo,Config,Input,Output>
Class DecoratorTap wraps a given Tap
instance, delegating all calls to the original.
Checkpoint
Pipe
.
Sub-classing this is optional if the aim is to simply attach relevant meta-info for use by a given application.
In order to pass any meta-info to a management service via the Property
annotation, a sub-class of DecoratorTap must be provided.
Field Summary | |
---|---|
protected MetaInfo |
metaInfo
|
protected Tap<Config,Input,Output> |
original
|
Constructor Summary | |
---|---|
DecoratorTap(MetaInfo metaInfo,
Tap<Config,Input,Output> original)
Creates a new Tap instance, wrapping the given Tap, and associates the given MetaInfo type with the wrapped Tap instance. |
|
DecoratorTap(Tap<Config,Input,Output> original)
Creates a new Tap instance, wrapping the given Tap, and associates the given MetaInfo type with the wrapped Tap instance. |
Method Summary | |
---|---|
boolean |
commitResource(Config conf)
Method commitResource allows the underlying resource to be notified when all write processing is successful so that any additional cleanup or processing may be completed. |
boolean |
createResource(Config conf)
Method createResource creates the underlying resource. |
boolean |
createResource(FlowProcess<Config> flowProcess)
Method createResource creates the underlying resource. |
boolean |
deleteResource(Config conf)
Method deleteResource deletes the resource represented by this instance. |
boolean |
deleteResource(FlowProcess<Config> flowProcess)
Method deleteResource deletes the resource represented by this instance. |
void |
flowConfInit(Flow<Config> flow)
Method flowInit allows this Tap instance to initialize itself in context of the given Flow instance. |
ConfigDef |
getConfigDef()
Returns a ConfigDef instance that allows for local properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
String |
getFullIdentifier(Config conf)
Method getFullIdentifier returns a fully qualified resource identifier. |
String |
getFullIdentifier(FlowProcess<Config> flowProcess)
Method getFullIdentifier returns a fully qualified resource identifier. |
String |
getIdentifier()
Method getIdentifier returns a String representing the resource this Tap instance represents. |
MetaInfo |
getMetaInfo()
|
long |
getModifiedTime(Config conf)
Method getModifiedTime returns the date this resource was last modified. |
long |
getModifiedTime(FlowProcess<Config> flowProcess)
Method getModifiedTime returns the date this resource was last modified. |
Tap<Config,Input,Output> |
getOriginal()
|
Scheme<Config,Input,Output,?,?> |
getScheme()
Method getScheme returns the scheme of this Tap object. |
Fields |
getSinkFields()
Method getSinkFields returns the sinkFields of this Tap object. |
SinkMode |
getSinkMode()
Method getSinkMode returns the SinkMode }of this Tap object. |
Fields |
getSourceFields()
Method getSourceFields returns the sourceFields of this Tap object. |
ConfigDef |
getStepConfigDef()
Returns a ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
String |
getTrace()
|
boolean |
hasConfigDef()
Returns true if there are properties in the configDef instance. |
boolean |
hasStepConfigDef()
Returns true if there are properties in the processConfigDef instance. |
boolean |
isEquivalentTo(FlowElement element)
|
boolean |
isKeep()
Method isKeep indicates whether the resource represented by this instance should be kept if it already exists when the Flow is started. |
boolean |
isReplace()
Method isReplace indicates whether the resource represented by this instance should be deleted if it already exists when the Flow is started. |
boolean |
isSink()
Method isSink returns true if this Tap instance can be used as a sink. |
boolean |
isSource()
Method isSource returns true if this Tap instance can be used as a source. |
boolean |
isTemporary()
Method isTemporary returns true if this Tap is temporary (used for intermediate results). |
boolean |
isUpdate()
Method isUpdate indicates whether the resource represented by this instance should be updated if it already exists. |
TupleEntryIterator |
openForRead(FlowProcess<Config> flowProcess)
Method openForRead opens the resource represented by this Tap instance for reading. |
TupleEntryIterator |
openForRead(FlowProcess<Config> flowProcess,
Input input)
Method openForRead opens the resource represented by this Tap instance for reading. |
TupleEntryCollector |
openForWrite(FlowProcess<Config> flowProcess)
Method openForWrite opens the resource represented by this Tap instance for writing. |
TupleEntryCollector |
openForWrite(FlowProcess<Config> flowProcess,
Output output)
Method openForWrite opens the resource represented by this Tap instance for writing. |
cascading.flow.planner.Scope |
outgoingScopeFor(Set<cascading.flow.planner.Scope> incomingScopes)
Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement. |
void |
presentSinkFields(FlowProcess<Config> flowProcess,
Fields fields)
|
void |
presentSourceFields(FlowProcess<Config> flowProcess,
Fields fields)
|
Fields |
resolveIncomingOperationArgumentFields(cascading.flow.planner.Scope incomingScope)
Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments. |
Fields |
resolveIncomingOperationPassThroughFields(cascading.flow.planner.Scope incomingScope)
Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple. |
boolean |
resourceExists(Config conf)
Method resourceExists returns true if the path represented by this instance exists. |
boolean |
resourceExists(FlowProcess<Config> flowProcess)
Method resourceExists returns true if the path represented by this instance exists. |
Fields |
retrieveSinkFields(FlowProcess<Config> flowProcess)
A hook for allowing a Scheme to lazily retrieve its sink fields. |
Fields |
retrieveSourceFields(FlowProcess<Config> flowProcess)
A hook for allowing a Scheme to lazily retrieve its source fields. |
boolean |
rollbackResource(Config conf)
Method rollbackResource allows the underlying resource to be notified when any write processing has failed or was stopped so that any cleanup may be started. |
protected void |
setMetaInfo(MetaInfo metaInfo)
|
protected void |
setOriginal(Tap<Config,Input,Output> original)
|
void |
sinkConfInit(FlowProcess<Config> flowProcess,
Config conf)
Method sinkConfInit initializes this instance as a sink. |
void |
sourceConfInit(FlowProcess<Config> flowProcess,
Config conf)
Method sourceConfInit initializes this instance as a source. |
String |
toString()
|
Methods inherited from class cascading.tap.Tap |
---|
equals, hashCode, id, setScheme, taps |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
protected MetaInfo metaInfo
protected Tap<Config,Input,Output> original
Constructor Detail |
---|
@ConstructorProperties(value={"metaInfo","original"}) public DecoratorTap(MetaInfo metaInfo, Tap<Config,Input,Output> original)
metaInfo
- meta-information about the current Taporiginal
- the decorated Tap instance@ConstructorProperties(value="original") public DecoratorTap(Tap<Config,Input,Output> original)
original
- the decorated Tap instanceMethod Detail |
---|
public MetaInfo getMetaInfo()
public Tap<Config,Input,Output> getOriginal()
protected void setOriginal(Tap<Config,Input,Output> original)
protected void setMetaInfo(MetaInfo metaInfo)
public Scheme<Config,Input,Output,?,?> getScheme()
Tap
getScheme
in class Tap<Config,Input,Output>
public String getTrace()
getTrace
in interface cascading.util.Traceable
getTrace
in class Tap<Config,Input,Output>
public void flowConfInit(Flow<Config> flow)
Tap
Flow
instance.
This method is guaranteed to be called before the Flow is started and the
FlowListener.onStarting(cascading.flow.Flow)
event is fired.
This method will be called once per Flow, and before Tap.sourceConfInit(cascading.flow.FlowProcess, Object)
and
Tap.sinkConfInit(cascading.flow.FlowProcess, Object)
methods.
flowConfInit
in class Tap<Config,Input,Output>
flow
- of type Flowpublic void sourceConfInit(FlowProcess<Config> flowProcess, Config conf)
Tap
Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method.
sourceConfInit
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessconf
- of type Configpublic void sinkConfInit(FlowProcess<Config> flowProcess, Config conf)
Tap
Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
Note this method will be called in context of this Tap being used as a traditional 'sink' and as a 'trap'.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method. If this Tap instance returns true for
Tap.isReplace()
, then Tap.deleteResource(Object)
will be called by the parent Flow.
sinkConfInit
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessconf
- of type Configpublic String getIdentifier()
Tap
getIdentifier
in class Tap<Config,Input,Output>
public Fields getSourceFields()
Tap
getSourceFields
in class Tap<Config,Input,Output>
public Fields getSinkFields()
Tap
getSinkFields
in class Tap<Config,Input,Output>
public TupleEntryIterator openForRead(FlowProcess<Config> flowProcess, Input input) throws IOException
Tap
input
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
input type and instantiate it before calling super.openForRead()
.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
openForRead
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessinput
- of type Input
IOException
- when the resource cannot be openedpublic TupleEntryIterator openForRead(FlowProcess<Config> flowProcess) throws IOException
Tap
TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
openForRead
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
IOException
- when the resource cannot be openedpublic TupleEntryCollector openForWrite(FlowProcess<Config> flowProcess, Output output) throws IOException
Tap
SinkMode
setting. If SinkMode is
SinkMode.REPLACE
, this call may fail. See Tap.openForWrite(cascading.flow.FlowProcess)
.
output
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
output type and instantiate it before calling super.openForWrite()
.
openForWrite
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessoutput
- of type Output
IOException
- when the resource cannot be openedpublic TupleEntryCollector openForWrite(FlowProcess<Config> flowProcess) throws IOException
Tap
SinkMode.REPLACE
settings. That is, if
SinkMode is set to SinkMode.REPLACE
the underlying resource will be deleted.
Note if SinkMode.UPDATE
is set, the resource will not be deleted.
openForWrite
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
IOException
- when the resource cannot be openedpublic cascading.flow.planner.Scope outgoingScopeFor(Set<cascading.flow.planner.Scope> incomingScopes)
FlowElement
outgoingScopeFor
in interface FlowElement
outgoingScopeFor
in class Tap<Config,Input,Output>
incomingScopes
- of type Setpublic Fields retrieveSourceFields(FlowProcess<Config> flowProcess)
Tap
retrieveSourceFields
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
public void presentSourceFields(FlowProcess<Config> flowProcess, Fields fields)
presentSourceFields
in class Tap<Config,Input,Output>
public Fields retrieveSinkFields(FlowProcess<Config> flowProcess)
Tap
retrieveSinkFields
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
public void presentSinkFields(FlowProcess<Config> flowProcess, Fields fields)
presentSinkFields
in class Tap<Config,Input,Output>
public Fields resolveIncomingOperationArgumentFields(cascading.flow.planner.Scope incomingScope)
FlowElement
resolveIncomingOperationArgumentFields
in interface FlowElement
resolveIncomingOperationArgumentFields
in class Tap<Config,Input,Output>
incomingScope
- of type Scope
public Fields resolveIncomingOperationPassThroughFields(cascading.flow.planner.Scope incomingScope)
FlowElement
resolveIncomingOperationPassThroughFields
in interface FlowElement
resolveIncomingOperationPassThroughFields
in class Tap<Config,Input,Output>
incomingScope
- of type Scope
public String getFullIdentifier(FlowProcess<Config> flowProcess)
Tap
getFullIdentifier
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
public String getFullIdentifier(Config conf)
Tap
getFullIdentifier
in class Tap<Config,Input,Output>
conf
- of type Config
public boolean createResource(FlowProcess<Config> flowProcess) throws IOException
Tap
createResource
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
IOException
- when there is an error making directoriespublic boolean createResource(Config conf) throws IOException
Tap
createResource
in class Tap<Config,Input,Output>
conf
- of type Config
IOException
- when there is an error making directoriespublic boolean deleteResource(FlowProcess<Config> flowProcess) throws IOException
Tap
deleteResource
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
IOException
- when the resource cannot be deletedpublic boolean deleteResource(Config conf) throws IOException
Tap
deleteResource
in class Tap<Config,Input,Output>
conf
- of type Config
IOException
- when the resource cannot be deletedpublic boolean commitResource(Config conf) throws IOException
Tap
Tap.rollbackResource(Object)
to handle cleanup in the face of failures.
This method is invoked once "client side" and not in the cluster, if any.
If other sink Tap instance in a given Flow fail on commitResource after called on this instance,
rollbackResource will not be called.
commitResource
in class Tap<Config,Input,Output>
conf
- of type Config
IOException
public boolean rollbackResource(Config conf) throws IOException
Tap
Tap.commitResource(Object)
to handle cleanup when the write has successfully completed.
This method is invoked once "client side" and not in the cluster, if any.
rollbackResource
in class Tap<Config,Input,Output>
conf
- of type Config
IOException
public boolean resourceExists(FlowProcess<Config> flowProcess) throws IOException
Tap
resourceExists
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
IOException
- when the status cannot be determinedpublic boolean resourceExists(Config conf) throws IOException
Tap
resourceExists
in class Tap<Config,Input,Output>
conf
- of type Config
IOException
- when the status cannot be determinedpublic long getModifiedTime(FlowProcess<Config> flowProcess) throws IOException
Tap
getModifiedTime
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcess
IOException
public long getModifiedTime(Config conf) throws IOException
Tap
getModifiedTime
in class Tap<Config,Input,Output>
conf
- of type Config
IOException
public SinkMode getSinkMode()
Tap
SinkMode
}of this Tap object.
getSinkMode
in class Tap<Config,Input,Output>
public boolean isKeep()
Tap
isKeep
in class Tap<Config,Input,Output>
public boolean isReplace()
Tap
isReplace
in class Tap<Config,Input,Output>
public boolean isUpdate()
Tap
Tap.createResource(Object)
, when the Flow is started.
isUpdate
in class Tap<Config,Input,Output>
public boolean isSink()
Tap
isSink
in class Tap<Config,Input,Output>
public boolean isSource()
Tap
isSource
in class Tap<Config,Input,Output>
public boolean isTemporary()
Tap
isTemporary
in class Tap<Config,Input,Output>
public ConfigDef getConfigDef()
Tap
ConfigDef
instance that allows for local properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the configDef will not show up in any Flow
or FlowStep
process
level configuration, but will override any of those values as seen by the current Tap instance method call where a
FlowProcess is provided except for the Tap.sourceConfInit(cascading.flow.FlowProcess, Object)
and
Tap.sinkConfInit(cascading.flow.FlowProcess, Object)
methods.
That is, the *confInit
methods are called before any ConfigDef is applied, so any values placed into
a ConfigDef instance will not be visible to them.
getConfigDef
in interface FlowElement
getConfigDef
in class Tap<Config,Input,Output>
public boolean hasConfigDef()
Tap
true
if there are properties in the configDef instance.
hasConfigDef
in interface FlowElement
hasConfigDef
in class Tap<Config,Input,Output>
public ConfigDef getStepConfigDef()
Tap
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowStep
(in Hadoop the MapReduce jobconf). Any value set in the
stepConfigDef will be overridden by the tap local #getConfigDef
instance.
Use this method to tweak properties in the process step this tap instance is planned into.
Note the *confInit
methods are called before any ConfigDef is applied, so any values placed into
a ConfigDef instance will not be visible to them.
getStepConfigDef
in interface FlowElement
getStepConfigDef
in class Tap<Config,Input,Output>
public boolean hasStepConfigDef()
Tap
true
if there are properties in the processConfigDef instance.
hasStepConfigDef
in interface FlowElement
hasStepConfigDef
in class Tap<Config,Input,Output>
public boolean isEquivalentTo(FlowElement element)
isEquivalentTo
in interface FlowElement
isEquivalentTo
in class Tap<Config,Input,Output>
public String toString()
toString
in class Tap<Config,Input,Output>
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |