cascading.scheme
Class TextDelimited

java.lang.Object
  cascading.scheme.Scheme
      cascading.scheme.TextLine
          cascading.scheme.TextDelimited

All Implemented Interfaces:

Serializable

public class TextDelimited
extends TextLine

Class TextDelimited is a sub-class of TextLine. It provides direct support for delimited text files, like TAB (\t) or COMMA (,) delimited files. It also optionally allows for quoted values.

TextDelimited may also be used to skip the "header" in a file, where the header is defined as the very first line in every input file. That is, if the byte offset of the current line from the input is zero (0), that line will be skipped.

By default headers are not skipped.

By default this Scheme is both strict and safe.

Strict meaning if a line of text does not parse into the expected number of fields, this class will throw a TapException. If strict is false, then Tuple will be returned with null values for the missing fields.

Safe meaning if a field cannot be coerced into an expected type, a null will be used for the value. If safe is false, a TapException will be thrown.

Also by default, quote strings are not searched for to improve processing speed. If a file is COMMA delimited but may have COMMA's in a value, the whole value should be surrounded by the quote string, typically double quotes (").

Note all empty fields in a line will be returned as null unless coerced into a new type.

This Scheme may source/sink Fields.ALL, when given on the constructor the new instance will automatically default to strict == false as the number of fields parsed are arbitrary or unknown. A type array may not be given either, so all values will be returned as Strings.

See Also:

TextLine, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class cascading.scheme.TextLine

TextLine.Compress

Field Summary

protected Pattern	cleanPattern Field cleanPattern
protected Pattern	escapePattern Field escapePattern
protected Pattern	splitPattern Field splitPattern

Fields inherited from class cascading.scheme.TextLine

DEFAULT_SOURCE_FIELDS

Constructor Summary

TextDelimited(Fields fields, boolean skipHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, boolean skipHeader, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter, boolean strict, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, boolean skipHeader, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, String quote)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, String quote, Class[] types)
Constructor TextDelimited creates a new TextDelimited instance.

TextDelimited(Fields fields, TextLine.Compress sinkCompression, String delimiter, String quote, Class[] types, boolean safe)
Constructor TextDelimited creates a new TextDelimited instance.

Method Summary

static Object[]	cleanSplit(Object[] split, Pattern cleanPattern, Pattern escapePattern, String quote) Method cleanSplit will return a quote free array of String values, the given split array will be updated in place.
static Pattern	createCleanPatternFor(String quote) Method createCleanPatternFor creates a regex Pattern for removing quote characters from a String.
static Pattern	createEscapePatternFor(String quote) Method createEscapePatternFor creates a regex Pattern cleaning quote escapes from a String.
static String[]	createSplit(String value, Pattern splitPattern, int numValues) Method createSplit will split the given value with the given splitPattern.
static Pattern	createSplitPatternFor(String delimiter, String quote) Method createSplitPatternFor creates a regex Pattern for splitting a line of text into its component parts using the given delimiter and quote Strings.
void	sink(TupleEntry tupleEntry, OutputCollector outputCollector) Method sink writes out the given Tuple instance to the outputCollector.
Tuple	source(Object key, Object value) Method source takes the given Hadoop key and value and returns a new Tuple instance.

Methods inherited from class cascading.scheme.TextLine

getSinkCompression, setSinkCompression, sinkInit, sourceInit

Methods inherited from class cascading.scheme.Scheme

equals, getNumSinkParts, getSinkFields, getSourceFields, getTrace, hashCode, isSink, isSource, isSymmetrical, isWriteDirect, setNumSinkParts, setSinkFields, setSourceFields, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

splitPattern

protected Pattern splitPattern

Field splitPattern

cleanPattern

protected Pattern cleanPattern

Field cleanPattern

escapePattern

protected Pattern escapePattern

Field escapePattern

Constructor Detail

TextDelimited

@ConstructorProperties(value={"fields","delimiter"})
public TextDelimited(Fields fields,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","delimiter"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","delimiter","types"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","delimiter","types"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                String delimiter,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                String delimiter,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","delimiter","quote"})
public TextDelimited(Fields fields,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","skipHeader","delimiter","quote"})
public TextDelimited(Fields fields,
                                                boolean skipHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

skipHeader - of type boolean

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","quote"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

quote - of type String

TextDelimited

public TextDelimited(Fields fields,
                     TextLine.Compress sinkCompression,
                     boolean skipHeader,
                     String delimiter,
                     String quote)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

quote - of type String

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","quote","types"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                String delimiter,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

quote - of type String

types - of type Class[]

safe - of type boolean

TextDelimited

@ConstructorProperties(value={"fields","sinkCompression","skipHeader","delimiter","strict","quote","types","safe"})
public TextDelimited(Fields fields,
                                                TextLine.Compress sinkCompression,
                                                boolean skipHeader,
                                                String delimiter,
                                                boolean strict,
                                                String quote,
                                                Class[] types,
                                                boolean safe)

Constructor TextDelimited creates a new TextDelimited instance.

Parameters:

fields - of type Fields

sinkCompression - of type Compress

skipHeader - of type boolean

delimiter - of type String

strict - of type boolean

quote - of type String

types - of type Class[]

safe - of type boolean

Method Detail

createEscapePatternFor

public static Pattern createEscapePatternFor(String quote)

Method createEscapePatternFor creates a regex Pattern cleaning quote escapes from a String.

If quote is null or empty, a null value will be returned;

Parameters:

quote - of type String

Returns:

Pattern

createCleanPatternFor

public static Pattern createCleanPatternFor(String quote)

Method createCleanPatternFor creates a regex Pattern for removing quote characters from a String.

If quote is null or empty, a null value will be returned;

Parameters:

quote - of type String

Returns:

Pattern

createSplitPatternFor

public static Pattern createSplitPatternFor(String delimiter,
                                            String quote)

Method createSplitPatternFor creates a regex Pattern for splitting a line of text into its component parts using the given delimiter and quote Strings. quote may be null.

Parameters:

delimiter - of type String

quote - of type String

Returns:

Pattern

source

public Tuple source(Object key,
                    Object value)

Description copied from class: Scheme

Method source takes the given Hadoop key and value and returns a new Tuple instance.

Overrides:

source in class TextLine

Parameters:

key - of type WritableComparable

value - of type Writable

Returns:

Tuple

createSplit

public static String[] createSplit(String value,
                                   Pattern splitPattern,
                                   int numValues)

Method createSplit will split the given value with the given splitPattern.

Parameters:

value - of type String

splitPattern - of type Pattern

numValues - of type int

Returns:

String[]

cleanSplit

public static Object[] cleanSplit(Object[] split,
                                  Pattern cleanPattern,
                                  Pattern escapePattern,
                                  String quote)

Method cleanSplit will return a quote free array of String values, the given split array will be updated in place.

If cleanPattern is null, quote cleaning will not be performed, but all empty String values will be replaces with a null value.

Parameters:

split - of type Object[]

cleanPattern - of type Pattern

escapePattern - of type Pattern

quote - of type String

Returns:

Object[] as a convenience

sink

public void sink(TupleEntry tupleEntry,
                 OutputCollector outputCollector)
          throws IOException

Description copied from class: Scheme

Method sink writes out the given Tuple instance to the outputCollector.

Overrides:

sink in class TextLine

outputCollector - of type OutputCollector @throws IOException when

Throws:

IOException