During runtime, Hadoop must be told which application jar file
should be pushed to the cluster. Typically, this is done via the Hadoop
Cascading offers a shorthand for configuring this parameter, demonstrated here:
Properties properties = new Properties(); // pass in the class name of your application // this will find the parent jar at runtime AppProps.setApplicationJarClass( properties, Main.class ); // ALTERNATIVELY ... // pass in the path to the parent jar AppProps.setApplicationJarPath( properties, pathToJar ); // pass properties to the connector FlowConnector flowConnector = new HadoopFlowConnector( properties );
Above we see two ways to set the same property - via the
setApplicationJarClass() method, and via the
setApplicationJarPath() method. One is based on
a Class name, and the other is based on a literal path.
The first method takes a Class object that owns the "main"
function for this application. The assumption here is that
Main.class is not located in a Java Jar that is stored in
lib folder of the application Jar. If it is,
that Jar is pushed to the cluster, not the parent application
The second method simply sets the path to the parent Class as a property.
In your application, only one of these methods needs to be called, but one of them must be called to properly configure Hadoop.
Example 4.1. Configuring the Application Jar with a JobConf
JobConf jobConf = new JobConf(); // pass in the class name of your application // this will find the parent jar at runtime jobConf.setJarByClass( Main.class ); // ALTERNATIVELY ... // pass in the path to the parent jar jobConf.setJar( pathToJar ); // build the properties object using jobConf as defaults Properties properties = AppProps.appProps() .setName( "sample-app" ) .setVersion( "1.2.3" ) .buildProperties( jobConf ); // pass properties to the connector FlowConnector flowConnector = new HadoopFlowConnector( properties );
Above we are starting with an existing Hadoop
JobConf instance and building a Properties object
with it as the default.
Copyright © 2007-2012 Concurrent, Inc. All Rights Reserved.