cascading.scheme.hadoop
Class TextDelimited

java.lang.Object
  cascading.scheme.Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],Object[]>
      cascading.scheme.hadoop.TextLine
          cascading.scheme.hadoop.TextDelimited

All Implemented Interfaces:

Serializable

public class TextDelimited
extends TextLine

Class TextDelimited is a sub-class of TextLine. It provides direct support for delimited text files, like TAB (\t) or COMMA (,) delimited files. It also optionally allows for quoted values.

TextDelimited may also be used to skip the "header" in a file, where the header is defined as the very first line in every input file. That is, if the byte offset of the current line from the input is zero (0), that line will be skipped.

It is assumed if sink/source fields is set to either Fields.ALL or Fields.UNKNOWN and skipHeader or hasHeader is true, the field names will be retrieved from the header of the file and used during planning. The header will parsed with the same rules as the body of the file.

By default headers are not skipped.

TextDelimited may also be used to write a "header" in a file. The fields names for the header are taken directly from the declared fields. Or if the declared fields are Fields.ALL or Fields.UNKNOWN, the resolved field names will be used, if any.

By default headers are not written.

If hasHeaders is set to true on a constructor, both skipHeader and writeHeader will be set to true.

By default this Scheme is both strict and safe.

Strict meaning if a line of text does not parse into the expected number of fields, this class will throw a TapException. If strict is false, then Tuple will be returned with null values for the missing fields.

Safe meaning if a field cannot be coerced into an expected type, a null will be used for the value. If safe is false, a TapException will be thrown.

Also by default, quote strings are not searched for to improve processing speed. If a file is COMMA delimited but may have COMMA's in a value, the whole value should be surrounded by the quote string, typically double quotes (").

Note all empty fields in a line will be returned as null unless coerced into a new type.

This Scheme may source/sink Fields.ALL, when given on the constructor the new instance will automatically default to strict == false as the number of fields parsed are arbitrary or unknown. A type array may not be given either, so all values will be returned as Strings.

By default, all text is encoded/decoded as UTF-8. This can be changed via the charsetName constructor argument.

See Also:

TextLine, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class cascading.scheme.hadoop.TextLine

TextLine.Compress

Field Summary

static String	DEFAULT_CHARSET
protected cascading.scheme.util.DelimitedParser	delimitedParser Field delimitedParser

Fields inherited from class cascading.scheme.hadoop.TextLine

DEFAULT_SOURCE_FIELDS

Constructor Summary

TextDelimited()
Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

TextDelimited(boolean hasHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

TextDelimited(boolean hasHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

TextDelimited(Fields fields)
Constructor TextDelimited creates a new TextDelimited instance with TAB as the default delimiter.

TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean hasHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean hasHeader, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote, Class[] types, boolean safe, String charsetName)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, boolean strict, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, boolean strict, String quote, Class[] types, boolean safe, String charsetName)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, boolean writeHeader, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, Class[] types, boolean safe, String charsetName)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, String quote, String charsetName)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(TextLine.Compress sinkCompression, boolean hasHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Method Summary

String	getDelimiter() Method getDelimiter returns the delimiter used to parse fields from the current line of text.
String	getQuote() Method getQuote returns the quote string, if any, used to encapsulate each field in a line to delimited text.
boolean	isSymmetrical() Method isSymmetrical returns true if the sink fields equal the source fields.
void	presentSinkFields(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, Tap tap, Fields fields) Method presentSinkFields is called after the planner is invoked and all fields are resolved.
void	presentSourceFields(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, Tap tap, Fields fields) Method presentSourceFields is called after the planner is invoked and all fields are resolved.
Fields	retrieveSourceFields(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, Tap tap) Method retrieveSourceFields notifies a Scheme when it is appropriate to dynamically update the fields it sources.
void	setSinkFields(Fields sinkFields) Method setSinkFields sets the sinkFields of this Scheme object.
void	setSourceFields(Fields sourceFields) Method setSourceFields sets the sourceFields of this Scheme object.
void	sink(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, SinkCall<Object[],org.apache.hadoop.mapred.OutputCollector> sinkCall) Method sink writes out the given Tuple found on SinkCall.getOutgoingEntry() to the SinkCall.getOutput().
void	sinkPrepare(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, SinkCall<Object[],org.apache.hadoop.mapred.OutputCollector> sinkCall) Method sinkPrepare is used to initialize resources needed during each call of Scheme.sink(cascading.flow.FlowProcess, SinkCall).
boolean	source(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess, SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall) Method source will read a new "record" or value from SourceCall.getInput() and populate the available Tuple via SourceCall.getIncomingEntry() and return true on success or false if no more values available.

Methods inherited from class cascading.scheme.hadoop.TextLine

getSinkCompression, makeEncodedString, setCharsetName, setSinkCompression, sinkConfInit, sourceCleanup, sourceConfInit, sourceHandleInput, sourcePrepare, verify

Methods inherited from class cascading.scheme.Scheme

equals, getNumSinkParts, getSinkFields, getSourceFields, getTrace, hashCode, isSink, isSource, presentSinkFieldsInternal, presentSourceFieldsInternal, retrieveSinkFields, setNumSinkParts, sinkCleanup, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_CHARSET

public static final String DEFAULT_CHARSET

See Also:

Constant Field Values

delimitedParser

protected final cascading.scheme.util.DelimitedParser delimitedParser

Field delimitedParser

Constructor Detail

TextDelimited

public TextDelimited()

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

TextDelimited

@ConstructorProperties(value={"hasHeader","delimiter"})
public TextDelimited(boolean hasHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

Parameters:

hasHeader -

delimiter -

TextDelimited

@ConstructorProperties(value={"hasHeader","delimiter","quote"})
public TextDelimited(boolean hasHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

Parameters:

hasHeader -

delimiter -

quote -

TextDelimited

@ConstructorProperties(value={"sinkCompression","hasHeader","delimiter","quote"})
public TextDelimited(TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

Parameters:

sinkCompression -

hasHeader -

delimiter -

quote -

TextDelimited

@ConstructorProperties(value="fields")
public TextDelimited(Fields fields)

Constructor TextDelimited creates a new TextDelimited instance with TAB as the default delimiter.

Parameters:

fields - of type Fields

TextDelimited

@ConstructorProperties(value={"fields","delimiter"})
public TextDelimited(Fields fields,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","hasHeader","delimiter"})
public TextDelimited(Fields fields,
                                                boolean hasHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

hasHeader - of type boolean

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","writeHeader","delimiter"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","delimiter","types"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","hasHeader","delimiter","types"})
public TextDelimited(Fields fields,
                                                boolean hasHeader,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

hasHeader - of type boolean

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","writeHeader","delimiter","types"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","hasHeader","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

hasHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","writeHeader","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","hasHeader","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

hasHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","hasHeader","delimiter","quote","types","safe","charsetName"})
public TextDelimited(Fields fields,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe,
                                                String charsetName)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

hasHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

charsetName - of type String

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","writeHeader","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","writeHeader","delimiter"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","writeHeader","delimiter","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","types","safe","charsetName"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                Class[] types,
                                                boolean safe,
                                                String charsetName)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

types - of type Class[]

safe - of type boolean

charsetName - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","writeHeader","delimiter","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","delimiter","quote"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","hasHeader","delimiter","quote"})
public TextDelimited(Fields fields,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

hasHeader - of type boolean

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","writeHeader","delimiter","quote"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","quote"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","quote"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","quote","charsetName"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote,
                                                String charsetName)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

quote - of type String

charsetName - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","writeHeader","delimiter","quote"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","writeHeader","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","hasHeader","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean hasHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

hasHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","writeHeader","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

writeHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","strict","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                boolean strict,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

strict - of type boolean

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","strict","quote","types","safe","charsetName"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                boolean writeHeader,
                                                String delimiter,
                                                boolean strict,
                                                String quote,
                                                Class[] types,
                                                boolean safe,
                                                String charsetName)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

strict - of type boolean

quote - of type String

types - of type Class[]

safe - of type boolean

charsetName - of type String

Method Detail

getDelimiter

public String getDelimiter()

Method getDelimiter returns the delimiter used to parse fields from the current line of text.

Returns:

a String

getQuote

public String getQuote()

Method getQuote returns the quote string, if any, used to encapsulate each field in a line to delimited text.

Returns:

a String

isSymmetrical

public boolean isSymmetrical()

Description copied from class: Scheme

Method isSymmetrical returns true if the sink fields equal the source fields. That is, this scheme sources the same fields as it sinks.

Overrides:

isSymmetrical in class Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],Object[]>

Returns:

the symmetrical (type boolean) of this Scheme object.

setSinkFields

public void setSinkFields(Fields sinkFields)

Description copied from class: Scheme

Method setSinkFields sets the sinkFields of this Scheme object.

Overrides:

setSinkFields in class Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],Object[]>

Parameters:

sinkFields - the sinkFields of this Scheme object.

setSourceFields

public void setSourceFields(Fields sourceFields)

Description copied from class: Scheme

Method setSourceFields sets the sourceFields of this Scheme object.

Overrides:

setSourceFields in class Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],Object[]>

Parameters:

sourceFields - the sourceFields of this Scheme object.

retrieveSourceFields

public Fields retrieveSourceFields(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                                   Tap tap)

Description copied from class: Scheme

Method retrieveSourceFields notifies a Scheme when it is appropriate to dynamically update the fields it sources. By default the current declared fields are returned.

The FlowProcess presents all known properties resolved by the current planner.

The instance is the parent Tap for this Scheme instance.

Overrides:

retrieveSourceFields in class Scheme<org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.mapred.OutputCollector,Object[],Object[]>

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

Returns:

Fields

presentSourceFields

public void presentSourceFields(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                                Tap tap,
                                Fields fields)

Description copied from class: Scheme

Method presentSourceFields is called after the planner is invoked and all fields are resolved. This method presents to the Scheme the actual source fields after any planner intervention.

This method is called after Scheme.retrieveSourceFields(cascading.flow.FlowProcess, cascading.tap.Tap).

Overrides:

presentSourceFields in class TextLine

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

fields - of type Fields

presentSinkFields

public void presentSinkFields(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                              Tap tap,
                              Fields fields)

Description copied from class: Scheme

Method presentSinkFields is called after the planner is invoked and all fields are resolved. This method presents to the Scheme the actual source fields after any planner intervention.

This method is called after Scheme.retrieveSinkFields(cascading.flow.FlowProcess, cascading.tap.Tap).

Overrides:

presentSinkFields in class TextLine

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

fields - of type Fields

source

public boolean source(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                      SourceCall<Object[],org.apache.hadoop.mapred.RecordReader> sourceCall)
               throws IOException

Description copied from class: Scheme

Method source will read a new "record" or value from SourceCall.getInput() and populate the available Tuple via SourceCall.getIncomingEntry() and return true on success or false if no more values available.

It's ok to set a new Tuple instance on the incomingEntry TupleEntry, or to simply re-use the existing instance.

Note this is only time it is safe to modify a Tuple instance handed over via a method call.

This method may optionally throw a TapException if it cannot process a particular instance of data. If the payload Tuple is set on the TapException, that Tuple will be written to any applicable failure trap Tap.

Overrides:

source in class TextLine

Parameters:

flowProcess - of type FlowProcess

sourceCall - of SourceCall

Returns:

returns true when a Tuple was successfully read

Throws:

IOException

sinkPrepare

public void sinkPrepare(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                        SinkCall<Object[],org.apache.hadoop.mapred.OutputCollector> sinkCall)
                 throws IOException

Description copied from class: Scheme

Method sinkPrepare is used to initialize resources needed during each call of Scheme.sink(cascading.flow.FlowProcess, SinkCall).

Be sure to place any initialized objects in the SinkContext so each instance will remain threadsafe.

Overrides:

sinkPrepare in class TextLine

Parameters:

flowProcess - of type FlowProcess

sinkCall - of type SinkCall

Throws:

IOException

sink

public void sink(FlowProcess<org.apache.hadoop.mapred.JobConf> flowProcess,
                 SinkCall<Object[],org.apache.hadoop.mapred.OutputCollector> sinkCall)
          throws IOException

Description copied from class: Scheme

Method sink writes out the given Tuple found on SinkCall.getOutgoingEntry() to the SinkCall.getOutput().

This method may optionally throw a TapException if it cannot process a particular instance of data. If the payload Tuple is set on the TapException, that Tuple will be written to any applicable failure trap Tap. If not set, the incoming Tuple will be written instead.

Overrides:

sink in class TextLine

Parameters:

flowProcess - of Process

sinkCall - of SinkCall

Throws:

IOException