Scalable Cobol Copybook Data Processing Using Cascading

This tutorial is an illustration on how to use Cascading for reading data generated by Mainframes. The application will work with offline data—​the application doesn’t connect directly to a mainframe computer, but assumes that the data has been exported from the mainframe to a platform where Cascading jobs are being executed. The following diagram shows in yellow the typical steps that the mainframe’s operator has to perform in order to prepare the data for processing by Cascading.

The code for this tutorial can be divided into two broad components:

The classes responsible for the low level translation were generated offline using Legstar and were imported into the project for completeness. In production applications, these classes will be imported as a jar dependency.

Let’s first see how we generated the parsers using Legstar

Step 1: Translate COBOL copybooks to XML schemas and Java beans

As explained earlier, the structure of the data is represented by Cobol Copybooks. To translate these structures into Java classes. For this tutorial, we make use of the open source Legstar project to do this translation. LegStar is a set of development tools and execution runtimes aimed at integrating legacy COBOL applications with Java and SOA technologies.

Legstar’s utility "cob2trans" can translate Cobol copybooks into XML Schemas. The generated XML schemas contain COBOL annotations that keep back references from each XML element to the originating COBOL item. The COBOL-annotated XML schemas are then processed via JAXB <link>. LegStar has a JAXB plugin that propagates the original COBOL annotations to the Java beans that JAXB generates. Once JAXB classes are produced, cob2trans invokes the Java compiler and then creates an additional set of classes called binding classes. These classes are key to the COBOL to Java transformation runtime performances—​they avoid the cost of reflection at runtime.

We have done this step offline, but have provided the resulting Java beans classes in the beans subpackage.

Step 2: Translate XML Schemas to Cascading Fields

Cascading models the data stream as a series of records, with each record containing one or more Fields. You can think of Fields as the columns in a database table. In step 1 we were able to parse the copybooks and create XML schemas and Java bean classes from it. We now need to convert the Cobol annotated XML schemas into Cascading Field classes. The source code for the translator classes is present in the "translate" package. We invoked the Cob2Fields translator and placed the resulting Field classes in the fields subpackage.

For instance, here’s one of the copybooks converted offline to a Fields class:

Now that we have the code generated for low level parsing of the copybooks, let’s use it to build a Cascading Scheme. In a typical application, the code generated by the steps performed so far will be bundled as a jar, and the application building and using the scheme will have a dependency on it. For the purposes of this tutorial, in order to keep things simple, we have provided the classes in their source code form.

Step 1: Create a Cascading Scheme

A Scheme in Cascading represents the format of the data an application is trying to read or write. Given that we are able to parse the cobol copybooks and translate them into Cascading Field classes, we are now in a position to develop our scheme.

Let’s examine the constructor of the Scheme class:

In the code shown above, we first instantiate a helper class CopybookConfig which is a container for all the beans and fields specific to our copybooks. For some other copybook, the generated bean and field classes will be different, and this container will hold other beans. You can easily modify this class for your particular use case. In the scheme’s constructor we use the CopybookConfig object to discover the fields specific to this copybook, and append them to the account key field.

The main processing logic of the CopybookScheme is encapsulated in the Source method which is responsible for accepting one input at a time, and converting it to a Cascading Tuple instance.

Step 2: Use the Scheme to Read Mainframe Data and Display Results

Now that we have the parser and the scheme, we are ready to wire everything together and create a simple app which reads a sample EBCIDC coded data and prints the values of some of the fields on the screen. Let’s take a look at the source code of the class app.Main.

In Cascading, data connectivity is provided with Taps. A Tap can read or write data according to the Scheme it is bound with. With the CopybookScheme, we first create an input tap to read the EBCDIC encoded copybook data:

Next, we create an output tap to print the values of some select fields on the screen. The sample data contains a lot of fields, but we will select and display the value of only some of them. The fields of interest are passed in to the constructor of the TextDelimited scheme, which is a class to print fields separated by a character separator. In this case, we use comma as the separator character. Note that while we used the CopybookScheme to read data, we are using the in-built Cascading scheme, TextDelimited for output.

Finally, we connect these two taps using a copy pipe, create the flow and execute it:

Executing this flow will read the data file which was generated randomly, and print the contents of some of its fields on the screen. The fields shown are the account key, followed by some transaction details. While we are only printing the field values on the screen, you can easily proceed with complex data manipulation tasks after you’re able to connect to the EBCDIC data as shown in this tutorial. Cascading has a wealth of in-built data processing primitives such as joins, group-by, etc which can express any custom data processing logic.