6.3 Failure Traps

A Flow with Failure Traps

Failure Traps are the same as a Tap sink (opposed to a source), except being bound to a particular tail element of the pipe assembly, traps can be bound to intermediate pipe assembly segments, like to a Stream Assertion.

Whenever an operation fails and throws an exception, and there is an associated trap, the offending Tuple will be saved to the resource specified by the trap Tap. This allows the job to continue processing without any data loss.

By design, clusters are hardware fault tolerant. Lose a node, the cluster continues working.

But software fault tolerance is a little different. Failure Traps provide a means for the processing to continue without losing track of the data that caused the fault. For high fidelity applications, this may not be so attractive, but low fidelity applications (like web page indexing) this can dramatically improve processing reliability.

Example 6.7. Setting Traps

// ...some useful pipes here

// name this pipe assembly segment
assembly = new Pipe( "assertions", assembly );

AssertNotNull notNull = new AssertNotNull();
assembly = new Each( assembly, AssertionLevel.STRICT, notNull );

AssertSizeEquals equals = new AssertSizeEquals( 6 );
assembly = new Each( assembly, AssertionLevel.STRICT, equals );

AssertMatchesAll matchesAll = new AssertMatchesAll( "(GET|HEAD|POST)" );
assembly = new Each( assembly, new Fields("method"),
                     AssertionLevel.STRICT, matchesAll );

// ...some more useful pipes here

Map<String,Tap> traps = new HashMap<String,Tap>();

traps.put( "assertions", trap );

FlowConnector flowConnector = new FlowConnector();
Flow flow =
  flowConnector.connect( "log-parser", source, sink, traps, assembly );

In the above example, we bind our trap Tap to the pipe assembly segment named "assertions". Note how we can name branches and segments by using a single Pipe instance and it applies to all subsequent Pipe instances.

[Note]Note

Traps are for exceptional cases, in the same way Java Exception handling is not for application flow control, thus traps are not a means to filter some data into other locations. Applications that need to filter good and bad data should do so explicitly.

Copyright © 2007-2008 Concurrent, Inc. All Rights Reserved.