Java Developers Guide to ETL with Cascading

Welcome to the Java Developer’s Guide to ETL, a tutorial that will take you through implementing the commonly-operated ETL tasks with Cascading.

The coding examples in each part refers to a commonly-used operation in ETL, and can be referred directly; you do not need to complete previous parts to use a given example.

Also, while we give references to Cascading Users Guide for the APIs used to implement the ETL tasks, this tutorial is not intended to serve as an introduction to Cascading. For that, we recommend that you follow the Cascading for the Impatient tutorial.

If you have a question or run into any problems send an email to the cascading-user-list.

This tutorial discusses the following topics, which include exercises and links to resource material:

Introduction to ETL

  • Discusses key evaluation criteria for deciding your ETL strategy

  • Explains basic Cascading concepts

  • Evaluates Cascading and Driven as a framework for implementing ETL applications

Prerequisites

  • Install Driven, Gradle, IDE and other software for running the tutorial

Part 1: File Copy

  • Simple ETL application that copies a file from one location to another

  • Filters data to exclude it from processing (bad data)

  • Specifies output format (tab delimited)

  • Bins data based on date ranges

Part 2: Filters

  • Separate unwanted data and store it to a different file for separate analysis

  • Implement in-line data quality checks

  • Perform different processing logic based on content

Part 3: Merge

  • Merge records from multiple input files using MultiSourceTap

Part 4: Count

  • Count number of unique records

  • Report "Top xx" occurrences

Part 5: Moving Averages

  • Implement advanced aggregation techniques using GroupBy() in Cascading

Part 6: Joins

  • Split pipe into different branches based on data content

  • Perform different processing on each branch

  • Join the branches using HashJoin() in Cascading

Next: