Cascading 3.2 User Guide - Introduction

1. Introduction

1.1. What Is Cascading?

2. Diving into the APIs

2.1. Anatomy of a Word-Count Application

3. Cascading Basic Concepts

3.1. Terminology

3.3. Pipes

3.4. Platforms

3.6. Sink Modes

3.7. Flows

4. Tuple Fields

4.1. Field Sets

5. Pipe Assemblies

5.1. Each and Every Pipes

5.2. Merge

5.3. GroupBy

5.4. CoGroup

5.5. HashJoin

6. Flows

6.1. Creating Flows from Pipe Assemblies

7. Cascades

7.1. Creating a Cascade

8. Configuring

8.1. Introduction

9. Local Platform

9.1. Building an Application

10. The Apache Hadoop Platforms

10.1. What is Apache Hadoop?

11. Apache Hadoop MapReduce Platform

11.1. Configuring Applications

11.3. Building

12. Apache Tez Platform

12.1. Configuring Applications

12.2. Building

13. Using and Developing Operations

13.1. Introduction

13.2. Functions

13.3. Filters

13.4. Aggregators

13.5. Buffers

14. Custom Taps and Schemes

14.1. Introduction

14.2. Custom Taps

15. Advanced Processing

15.1. SubAssemblies

16. Built-In Operations

16.1. Identity Function

16.9. Assertions

16.11. Buffers

17. Built-in SubAssemblies

17.1. Optimized Aggregations

18. Cascading Best Practices

18.1. Unit Testing

19. Extending Cascading

19.1. Scripting

20. Cookbook: Code Examples of Cascading Idioms

20.1. Tuples and Fields

20.5. API Usage

21. The Cascading Process Planner

21.1. FlowConnector

21.3. RuleRegistry


What Is Cascading?

Cascading is a data processing API and processing planner used for defining, sharing, and executing data-oriented applications. These applications can execute on a single computing node or distributed computing platform with minimal code changes.

Cascading applications can be written in a way that allows the developer to create data-parallel applications that minimize the coupling to any specific computing platform. Fewer platform dependencies optimizes portability.

On a single node, Cascading’s "local mode" can be used to efficiently test code and process local files before being deployed on a cluster.

On a distributed computing cluster, Cascading natively supports the Apache Hadoop platform. Specific planners for both MapReduce and the Apache Tez directed acyclic graphs (DAGs) are provided.

Additionally, Cascading can support new platforms as they emerge so business logic will not need to be rewritten in order to leverage state-of-the-art technologies. Visit for announcements and details on newly added platforms.

Cascading is open-source under the Apache 2.0 License.

Another Perspective

Another way to look at Cascading is to consider the typical stack behind running a query with syntax like SQL:

  1. The query syntax is parsed to an abstract syntax tree (AST).

  2. The AST is logically reduced to an intermediate model. During this processing, many logical optimizations can be applied.

  3. The intermediate model is translated into an executable model. Much of the processing between the intermediate and executable models applies physical optimizations.

Cascading provides both the intermediate model, and a planner that will translate that model into a physical executable model. The intermediate model is presented as a Java API, and the planner is both modifiable and pluggable.

Why Use Cascading?

Cascading was developed to allow organizations to rapidly develop complex data processing applications. The need for Cascading is typically driven by one or more of four cases:

  • Increasing data size

  • Increasing computation requirements

  • Increasing complexity in data centers

  • Increasing need for accountability and manageability

Cascading applications scale to load seamlessly. The business logic for small data that may become big data does not require changes to the application logic.

Cascading provides many core data processing primitives and operations. Business logic can be coded directly against the Cascading APIs, or higher order frameworks or languages can be created on top. These frameworks or languages can be used to improve developer or analyst productivity for a specific use-case or type of enterprise.

Cascading as a platform allows for integration and business logic to remain decoupled until runtime so that different storage platforms can be bound to the logic on demand. New platforms can be leveraged during computation as business needs or infrastructure resources change.

Cascading can be managed. Besides a number of core features that allow for Cascading applications to be easily operationalized, Cascading works with commercial tools (like Driven) so that applications can be monitored in real-time. Developers gain in-depth understanding of application behavior at runtime in order to improve performance and reliability. Operations teams that are responsible for managing business processes can easily monitor application status.

The Cascading Philosophy

Cascading was designed to be extensible, to behave deterministically, and to fail fast where possible.

Many default features and behaviors of Cascading can be replaced or overridden. Where the Cascading User Guide does not provide guidance, review the Javadoc for APIs and properties that can be used or modified.

The execution plans created by the Cascading planner are both intuitive and stable:

  • As processing logic is added during application development, the resulting plan is only an incremental increase in complexity and is proportional to the changes that were just applied. After becoming familiar with Cascading, you can easily understand the implication of adding a new CoGroup to the assembly and how it will behave on the cluster.

  • Every execution plan is the same when there are no code changes. Across versions of Cascading, any likely behavioral differences are documented (if not accompanied by a way to revert the behavior).

  • The Cascading planner creates all units of work up front. The planner verifies all dependencies are met from available resources like sources and sinks, down to the field level. If a downstream operation requires a specific field (for example, "zipcode"), the planner guarantees it is available upstream from a data source or operation.

Many of these ideals seem obvious, but many systems regenerate plans on the fly or attempt shortcuts during execution. Even if these methods of other systems are successful, there is no certainty that they result in improved runtime performance.

It is not acceptable to fail part way, in a non-deterministic (non-repeatable) fashion, for applications that may run hours or days at a stretch.

Who Are the Users?

Cascading users typically fall into three roles:

The application executor is a person (for example, a developer or analyst) or process (for example, a cron job) that runs a data processing application on a given cluster. This is typically done via the command line, using a prepackaged JAR file compiled against the Cascading libraries. The application can accept command-line parameters for customization during execution.

The process assembler assembles data processing workflows into unique applications. This work is generally a development task that involves chaining together operations to act on input data sets so that they produce output data sets. Development can be done with the default Java Cascading API, the Fluid API, or with a scripting language such as Scala, Clojure, Groovy, JRuby, Jython. Cascading also supports domain-specific languages that are implemented in the scripting languages.

The operation developer writes individual functions or operations (typically in Java) or reusable subassemblies that act on the data that passes through the data processing workflow. A trivial example would be a parser that takes a string and converts it to an integer. Operations are equivalent to Java functions in the sense that they take input arguments and return data. Operations can execute at any granularity, from simply parsing a string to performing complex procedures on the argument data using third-party libraries. Cascading provides many prebuilt operations.

Each of the three roles can be filled by a developer. But in some organizations non-developers might run ad-hoc applications or build production processes because Cascading supports clean separation of the responsibilities that are entailed in the three roles.

As of Cascading 3.0, two new roles have emerged:

The platform optimizer improves execution runtime or resource utilization. This role can be responsible for creating workload-specific query-plan rules that target specific improvements. Another responsibility of the platform optimizer could be to add new core primitives that can be leveraged in application logic.

The platform developer ports Cascading to new computing platforms. This is an emerging API, but already proven to be very robust and powerful. As business needs change and new technologies emerge, a developer can create bindings to these new technologies allowing existing investments in the Cascading API and broader ecosystem to be leveraged.