Cascading 3.0 User Guide - Cascading Best Practices

Discrete testing of each Operation, pipe assembly, and application is a must.

The cascading.CascadingTestCase provides a number of static helper methods.

When testing custom Operations, use the invokeFunction(), invokeFilter(), invokeAggregator(), and invokeBuffer() methods.

When testing Flows, use the validateLength() methods. There are quite a few of them, and collectively they offer great flexibility. All of them read the sink tap, validate that it is the correct length and has the correct Tuple size, and check to see whether the values match a given regular expression pattern.

It is also possible to write tests that are independent of the underlying platform. Any unit test should subclass cascading.PlatformTestCase located in the cascading-platform-x.y.z-tests.jar file.

Any platform to be tested against should be added to the CLASSPATH as well. PlatformTestCase searches the CLASSPATH for all available platforms and runs each test on the subclass against each platform found.

See the Cascading platform unit tests for examples, all of which adhere to the naming convention of *PlatformTest.java.

Although using one large Flow may result in slightly more efficient performance, it’s advisable to use a more modular and flexible approach. Such an approach involves creating medium-sized Flows with well-defined responsibilities and passing all the resulting interdependent Flows to a Cascade to sequence and execute as a single unit.

Similarly, using the TextDelimited Scheme (or any custom format designed for long-term archival) between Flow instances allows you to hand off intermediate data to other systems for reporting or QA purposes, incurring a minimal performance penalty while remaining compatible with other tools.

When developing your applications, use SubAssembly subclasses, not "factory" methods. The resulting code is much easier to read and test.

SubAssemblies provide a very convenient means to co-locate similar or related responsibilities into a single place. For example, it’s simple to use a ParsingSubAssembly and a RulesSubAssembly, where the first is responsible solely for parsing incoming Tuple streams (log files for example), and the second applies rules to decide whether a given Tuple should be discarded or marked as bad.

Additionally, in your unit tests you can create a TestAssertionsSubAssembly that simply inlines various ValueAssertions and GroupAssertions. The practice of inlining assertions directly in your SubAssemblies is also important, but sometimes it makes sense to have more tests outside of the business logic.

There are a few Operations in Cascading (e.g., ExpressionFunction and ExpressionFilter) that compile and apply Java expressions as needed during processing. In these expressions, Operation argument field names are used as variable names in the expression.

For this reason, be sure to create field names without characters that cause compilation errors if they are used in an expression. For example, "first-name" is a valid field name for use with Cascading, but might result in the expression first-name.trim(), which would cause a compilation error.

The FlowConnector sometimes fails when attempting to plan a Flow. If the error message given by PlannerException is vague, use the PlannerException.writeDOT() method to export a representation of the internal pipe assembly.

DOT files can be opened with Graphviz or OmniGraffle. The plans in these files are only partial, but the graphic renderings show you where the Cascading planner failed.

Note that you can also create a DOT file from a Flow by using Flow.writeDOT() to better understand how the planner has modified your business logic.

If the above methods do not provide insight, the new Cascading 3.0 planner has a much improved debugging framework.

When running tests, set the following:

If you are on Mac OS X and have installed Graphviz, DOT files can be converted to PDF automatically. To enable this feature, set:

Optionally, for stand-alone applications, statistics and tracing can be enabled selectively when the following properties are passed to the FlowConnector:

When joining two streams via a CoGroup pipe, try to put the larger stream in the leftmost argument to the CoGroup.

The reason for this is that joining multiple streams requires some accumulation of values before the join operator can begin, but the leftmost stream is not accumulated. Consequently, this technique should improve the performance of most joins.

When creating complex assemblies, it’s safe to embed Debug Operations at appropriate debug levels as needed. To avoid wasting resources, use the planner to remove the Debug>> Operations at runtime for production and staging runs.

Corrupt data often enters raw data streams. For instance, bad content may be fetched from the web via a crawler upstream, or a bug may have leaked into a browser widget that sends user behavior information back for analysis. Whatever the cause, it’s a good practice to define a set of rules for identifying and managing questionable records.

It is tempting to simply throw an exception and have a Trap capture the offending Tuple. However, Traps were not designed as a filtering mechanism, and consequently much valuable information is lost when Traps are used in this situation.

Instead of Traps, use Filters. Create a SubAssembly that applies rules to the stream by setting a binary field that marks the Tuple as good or bad. After all the rules are applied, split the stream based on the value of the good or bad Boolean value. Consider setting a reason field that states why the Tuple was marked bad.

When creating custom Operations (Function, Filter, Aggregator, or Buffer), do not store the Operation state in class fields.

For example, if implementing an Aggregator as a custom "counter," do not create a field named "count" and increment it on every Aggregator.aggregate() call.

There is no guarantee that your Operation will be called from a single thread in a JVM. Also, future versions of Hadoop or Cascading local mode might execute the same Operation from multiple threads.

Instead of interspersing String field names throughout the code, create an interface that holds a constant value for each field name:

Using the Fields class, instead of String, allows for building more complex constants:

When in doubt, look at the Cascading source code. If something is not documented in this User Guide or Javadoc and it’s a feature of Cascading, the feature source code or unit tests will give you clear instructions on what to do or expect.