Cascading 3.0 User Guide - Introduction

Cascading is a data processing API and processing planner used for defining, sharing, and executing data-oriented applications. These applications can execute on a single computing node or distributed computing platform with minimal code changes.

Cascading applications can be written in a way that allows the developer to create data-parallel applications that minimize the coupling to any specific computing platform. Fewer platform dependencies optimizes portability.

On a single node, Cascading’s "local mode" can be used to efficiently test code and process local files before being deployed on a cluster.

On a distributed computing cluster, Cascading natively supports the Apache Hadoop platform. Specific planners for both MapReduce and the Apache Tez directed acyclic graphs (DAGs) are provided.

Additionally, Cascading can support new platforms as they emerge so business logic will not need to be rewritten in order to leverage state-of-the-art technologies.

Another way to look at Cascading is to consider the typical stack behind running a query with syntax like SQL:

Cascading provides both the intermediate model, and a planner that will translate that model into a physical executable model. The intermediate model is presented as a Java API, and the planner is both modifiable and pluggable.

Cascading was developed to allow organizations to rapidly develop complex data processing applications. The need for Cascading is typically driven by one or more of four cases:

Cascading applications scale to load seamlessly. The business logic for small data that may become big data does not require changes to the application logic.

Cascading provides many core data processing primitives and operations. Business logic can be coded directly against the Cascading APIs, or higher order frameworks or languages can be created on top. These frameworks or languages can be used to improve developer or analyst productivity for a specific use-case or type of enterprise.

Cascading as a platform allows for integration and business logic to remain decoupled until runtime so that different storage platforms can be bound to the logic on demand. New platforms can be leveraged during computation as business needs or infrastructure resources change.

Cascading can be managed. Besides a number of core features that allow for Cascading applications to be easily operationalized, Cascading works with commercial tools so that applications can be monitored in real-time. Developers gain in-depth understanding of application behavior at runtime in order to improve performance and reliability. Operations teams that are responsible for managing business processes can easily monitor application status.

Cascading was designed to be extensible, to behave deterministically, and to fail fast where possible.

Many default features and behaviors of Cascading can be replaced or overridden. Where the Cascading User Guide does not provide guidance, review the Javadoc for APIs and properties that can be used or modified.

The execution plans created by the Cascading planner are both intuitive and stable:

Many of these ideals seem obvious, but many systems regenerate plans on the fly or attempt shortcuts during execution. Even if these methods of other systems are successful, there is no certainty that they result in improved runtime performance.

It is not acceptable to fail part way, in a non-deterministic (non-repeatable) fashion, for applications that may run hours or days at a stretch.

Cascading users typically fall into three roles:

The application executor is a person (for example, a developer or analyst) or process (for example, a cron job) that runs a data processing application on a given cluster. This is typically done via the command line, using a prepackaged JAR file compiled against the Cascading libraries. The application can accept command-line parameters for customization during execution.

The process assembler assembles data processing workflows into unique applications. This work is generally a development task that involves chaining together operations to act on input data sets so that they produce output data sets. Development can be done with the default Java Cascading API, the Fluid API, or with a scripting language such as Scala, Clojure, Groovy, JRuby, Jython. Cascading also supports domain-specific languages that are implemented in the scripting languages.

The operation developer writes individual functions or operations (typically in Java) or reusable subassemblies that act on the data that passes through the data processing workflow. A trivial example would be a parser that takes a string and converts it to an integer. Operations are equivalent to Java functions in the sense that they take input arguments and return data. Operations can execute at any granularity, from simply parsing a string to performing complex procedures on the argument data using third-party libraries. Cascading provides many prebuilt operations.

Each of the three roles can be filled by a developer. But in some organizations non-developers might run ad-hoc applications or build production processes because Cascading supports clean separation of the responsibilities that are entailed in the three roles.

As of Cascading 3.0, two new roles have emerged:

The platform optimizer improves execution runtime or resource utilization. This role can be responsible for creating workload-specific query-plan rules that target specific improvements. Another responsibility of the platform optimizer could be to add new core primitives that can be leveraged in application logic.

The platform developer ports Cascading to new computing platforms. This is an emerging API, but already proven to be very robust and powerful. As business needs change and new technologies emerge, a developer can create bindings to these new technologies allowing existing investments in the Cascading API and broader ecosystem to be leveraged.