Cascading 3.0 User Guide - Local Platform
Local Platform
Building an Application
The Cascading local mode has no special requirements for building outside the requirement for any Java application to be executed from the command line. However, there are two top-level dependencies that should be added to the build file:
- cascading-core-3.x.y.jar
-
This JAR contains the Cascading Core class files.
- cascading-local-3.x.y.jar
-
This JAR contains the Cascading local-mode class files.
Executing an Application
After completing a build of the application’s "main" class, the application can be run like any other Java-based command-line application.
Troubleshooting and Debugging
IDE debugging and testing in Cascading local mode, unlike Cascading on other platforms, is straightforward as all the processing happens in the local JVM and in local memory. Therefore, the first recommendation for debugging Cascading applications on a given platform is to first write tests that run in Cascading local mode.
| Because Cascading local mode runs entirely in memory, large data sets may cause an OutOfMemoryException. Also, be sure to adjust the Java runtime memory settings. |
In addition to using an IDE debugger, you can use two Cascading features to help sort out runtime issues.
One feature is the Debug filter. Best practice is to sprinkle Debug operators (see Debug Function) in the pipe assembly and rely on the planner to remove them at runtime by setting a DebugLevel.
Debug can only print to the local console via standard output or standard error. This print limitation makes it harder to use Debug on distributed platforms, as operations do not execute locally but on the cluster side. Debug provides the option to print the current field names, and a prefix can be set to help distinguish between instances of the Debug operation.
Additionally, the actual execution plan for a Flow can be written (and visualized) via the Flow.writeDOT() method. DOT files are simply text representations of graph data and can be read by tools like Graphviz and OmniGraffle.
In Cascading local mode, these execution plans are exactly as the pipe assemblies were coded, except the subassemblies are unwound and the field names across the Flow are resolved by the planner. In other words, Fields.ALL and other wild cards are converted to the actual field names or ordinals.
If the connect() method on the current FlowConnector fails, the resulting PlannerException has a writeDOT() method that shows the progress of the current planner.
For planner-related errors that appear during runtime when executing a Flow, see the chapter on the Cascading Process Planner.