Cascading for the Impatient, Part 6

At first glance, the notion of test-driven development (TDD) might seem a bit antithetical in the context of Big Data. After all, TDD is all about short development cycles, writing automated test cases which are intended to fail, and lots of refactoring. Those descriptions would not appear to fit with batch jobs involving terabytes of data and huge clusters running apps that take days to complete.

Stated in a different way, according to Kent Beck, TDD encourages simple designs and inspires confidence.'' That statement does actually fit well with Cascading. The API is intended to provide simple design patterns for working with data - GroupBy, Join, Count, Regex, Filter - so that the need for writing custom functions becomes relatively rare. That speaks to encouraging simple designs directly. The practice in Cascading of modeling business process and orchestrating MapReduce workflows - that speaks to inspiring confidence'' in a big way.

So now we will let the cat out of the bag for a little secret…​ Working with unstructured data at scale has been shown to be quite valuable (Google, Amazon, LinkedIn, Twitter, etc.) however most of the 'heavy lifting' which we perform in MapReduce workflows is essentially cleaning up data. DJ Patil explained this point quite eloquently in Data Jujitsu: ``It’s impossible to overstress this: 80% of the work in any data project is in cleaning the data…​ Work done up front in getting clean data will be amply repaid over the course of the project.''

Cleaning up the data allows for subsequent use of sampling techniques, dimensional reduction, and other practices which help alleviate some of the bottlenecks which might otherwise be encountered in Big Data. In other words, there are great use cases for formalisms which help demonstrate that ``dirty" data at scale has been cleaned up. Those turn out to be quite valuable in practice.

However, TDD practices tend to be based on unit tests or mocks…​ how does one write a quick unit test for a Godzilla-sized dataset?

The short answer is: you don’t. However, you can greatly reduce the need for writing unit test coverage by limiting the amount of custom code required. Hopefully we have shown that aspect of Cascading by now. Beyond that aspect, you can use sampling techniques to quantify the confidence for an app running correctly. You can also define system tests at scale in relatively simple ways. Furthermore, you can define contingencies for what to do when assumptions fail…​ as they inevitably do, at scale.

Let’s discuss sampling…​ generally speaking, large MapReduce workflows are relatively opaque processes which are difficult to observe. However, Cascading provides two techniques for observing portions of a workflow. One very simple approach is to insert a Debug into a pipe, to see the tuple values passing through a particular part of a workflow. Debug output goes to the log instead of a file, but it can be turned off, e.g., with a command line option. If the data is large, one can use a Sample filter to sample the tuple values which get written to the log.

Another approach is to use a Checkpoint, which forces intermediate data to be written out to HDFS. This may also become important for performance reasons, i.e., forcing results to disk to avoid recomputing – e.g., when there are multiple uses for the output of a pipe downstream such as with the right side of a HashJoin. Sampling may be performed either before (like with Debug) or after the data gets persisted to HDFS.

Next, let’s talk about system tests. Cascading include support for stream assertions. These provide mechanisms for asserting that the values in a tuple stream meet certain criteria - similar to the assert keyword in Java, or an assertNotNull() in a unit test. We can assert patterns strictly as unit tests during development, then run testing against regression data. For performance reasons, we might use command line options to turn off assertions in production. Or keep them, if a use case requires that level of guarantees.

Lastly, what to do when assumptions fail? One lesson of working with data at scale is that the best assumptions will inevitably fail. Unexpected things happen, and 80% of the work will be cleaning up problems. Cascading defines failure traps which capture data that causes an Operation to fail, e.g., throw an Exception. For example, perhaps 99% of the cases in your log files can be rolled up into a set of standard reports…​ but 1% requires manual review. Great, process the 99% which work and shunt the 1% failure cases into a special file, marked for manual review. Keep in mind, however, that traps are intended for handling exceptional cases. If you know in advance how to categorize good vs. bad data, then use a filter instead of a trap.

Meanwhile, a conceptual diagram for this implementation of TF-IDF in Cascading is shown as:

All code and data is in the part6 subdirectory of the github project.

Let’s add a unit test and show how that works into this example. In the Gradle build script build.gradle we need to modify the compile task to include JUnit and other testing dependencies:

Then we will add a test task:

A little restructuring of the source directories is required see our GitHub code repo, where it iss all set up property. Then we add a unit test for our custom function to 'scrub' tokens, which was created in Part 3. This goes into a new class ScrubTest.java:

This is a particularly good place for a unit test. Scrubbing tokens is a likely point at which edge cases get encountered at scale. In practice, you’d probably want to define even more unit tests.

Next, going back to the Main.java module, let’s add sink taps for writing out trapped data and checkpointed data:

Next we’ll modify the head of the existing pipe assembly for TF-IDF to incorporate a Stream Assertion. We use an AssertMatches to define the expected pattern for input data. Then we apply AssertionLevel.STRICT to force validation of the data:

Next we’ll add a Debug and DebugLevel.VERBOSE to the D branch, to trace the tuple values in the flow there:

Next we’ll add a Checkpoint after the join of the DF and D branches. That forces the tuples at this point in the workflow to be persisted to HDFS:

Next we have a relatively more complex set of taps to connect in the FlowDef, to include output data for TDD-related features:

Last, we’ll specify the verbosity level for the debug trace, and the strictness level for the stream assertion:

Modify the Main method to make those changes, then build a JAR file. You should be good to go. For those keeping score, the resulting physical plan in MapReduce for Part 6 now uses twelve mappers and nine reducers. In other words, we added one mapper as the overhead for gaining lots of test features.

The diagram for the Cascading flow will be in the dot/ subdirectory after the app runs. Here we have annotated it to show where the mapper and reducer phases are running, and also the sections which were added since Part 5:

To build the sample app from the command line use:

Running this version is as easy as:

The output log should include a warning, based on the stream assertion, which looks like this:

That is expected behavior. We directed the API to show warning when stream assertions failed. The data which caused this warning will get trapped.

Not too far after that point in the log, there should be debug output which looks like the following:

plus several more lines. That is the result of our debug trace.

Output text gets stored in the partition file output/tfidf which you can then verify:

Notice the data tuple output/trap:

That did not match the regex doc\\d+\\s.* which was specified by the stream assertion.

Here’s a log file from our run of the sample app, part 6. If your run looks terribly different, something is probably not set up correctly.

To run this same app on the Amazon AWS Elastic MapReduce service, based on their command line interface, use the following commands. Be sure to replace temp.cascading.org with your own S3 bucket name:

If you have not installed the Driven plugin, but would like to explore an instance of a historical run of Part 6, visit this link

That’s all folks!