TextDelimited (Cascading 2.0.8-${cascading.build.number})

Overview

Package

Class

Tree

Deprecated

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

cascading.scheme.local
Class TextDelimited

java.lang.Object
  cascading.scheme.Scheme<Properties,InputStream,OutputStream,LineNumberReader,PrintWriter>
      cascading.scheme.local.TextDelimited

All Implemented Interfaces:: Serializable

public class TextDelimited
extends Scheme<Properties,InputStream,OutputStream,LineNumberReader,PrintWriter>
extends Scheme<Properties,InputStream,OutputStream,LineNumberReader,PrintWriter>

Class TextDelimited provides direct support for delimited text files, like TAB (\t) or COMMA (,) delimited files. It also optionally allows for quoted values.

TextDelimited may also be used to skip the "header" in a file, where the header is defined as the very first line in every input file. That is, if the byte offset of the current line from the input is zero (0), that line will be skipped.

It is assumed if sink/source fields is set to either Fields.ALL or Fields.UNKNOWN and skipHeader or hasHeader is true, the field names will be retrieved from the header of the file and used during planning. The header will parsed with the same rules as the body of the file.

By default headers are not skipped.

TextDelimited may also be used to write a "header" in a file. The fields names for the header are taken directly from the declared fields. Or if the declared fields are Fields.ALL or Fields.UNKNOWN, the resolved field names will be used, if any.

By default headers are not written.

If hasHeaders is set to true on a constructor, both skipHeader and writeHeader will be set to true.

By default this Scheme is both strict and safe.

Strict meaning if a line of text does not parse into the expected number of fields, this class will throw a TapException. If strict is false, then Tuple will be returned with null values for the missing fields.

Safe meaning if a field cannot be coerced into an expected type, a null will be used for the value. If safe is false, a TapException will be thrown.

Also by default, quote strings are not searched for to improve processing speed. If a file is COMMA delimited but may have COMMA's in a value, the whole value should be surrounded by the quote string, typically double quotes (").

Note all empty fields in a line will be returned as null unless coerced into a new type.

This Scheme may source/sink Fields.ALL, when given on the constructor the new instance will automatically default to strict == false as the number of fields parsed are arbitrary or unknown. A type array may not be given either, so all values will be returned as Strings.

See Also:: TextLine, Serialized Form

Constructor Summary
`TextDelimited()` Constructor TextDelimited creates a new TextDelimited instance sourcing `Fields.UNKNOWN`, sinking `Fields.ALL` and using TAB as the default delimiter.
`TextDelimited(boolean hasHeader, String delimiter)` Constructor TextDelimited creates a new TextDelimited instance sourcing `Fields.UNKNOWN`, sinking `Fields.ALL` and using TAB as the default delimiter.
`TextDelimited(boolean hasHeader, String delimiter, String quote)` Constructor TextDelimited creates a new TextDelimited instance sourcing `Fields.UNKNOWN`, sinking `Fields.ALL` and using TAB as the default delimiter.
`TextDelimited(Fields fields)` Constructor TextDelimited creates a new TextDelimited instance with TAB as the default delimiter.
`TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, boolean strict, String quote, Class[] types, boolean safe)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, Class[] types)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, String quote, Class[] types)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean skipHeader, boolean writeHeader, String delimiter, String quote, Class[] types, boolean safe)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean hasHeader, String delimiter)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean hasHeader, String delimiter, Class[] types)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote, Class[] types)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, boolean hasHeader, String delimiter, String quote, Class[] types, boolean safe)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, String delimiter)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, String delimiter, Class[] types)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, String delimiter, String quote)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, String delimiter, String quote, Class[] types)` Constructor TextDelimited creates a new TextDelimited instance.
`TextDelimited(Fields fields, String delimiter, String quote, Class[] types, boolean safe)` Constructor TextDelimited creates a new TextDelimited instance.

Method Summary
`LineNumberReader`	`createInput(InputStream inputStream)`
`PrintWriter`	`createOutput(OutputStream outputStream)`
`boolean`	`isSymmetrical()` Method isSymmetrical returns `true` if the sink fields equal the source fields.
`void`	`presentSinkFields(FlowProcess<Properties> flowProcess, Tap tap, Fields fields)` Method presentSinkFields is called after the planner is invoked and all fields are resolved.
`void`	`presentSourceFields(FlowProcess<Properties> process, Tap tap, Fields fields)` Method presentSourceFields is called after the planner is invoked and all fields are resolved.
`Fields`	`retrieveSourceFields(FlowProcess<Properties> process, Tap tap)` Method retrieveSourceFields notifies a Scheme when it is appropriate to dynamically update the fields it sources.
`void`	`sink(FlowProcess<Properties> flowProcess, SinkCall<PrintWriter,OutputStream> sinkCall)` Method sink writes out the given `Tuple` found on `SinkCall.getOutgoingEntry()` to the `SinkCall.getOutput()`.
`void`	`sinkCleanup(FlowProcess<Properties> flowProcess, SinkCall<PrintWriter,OutputStream> sinkCall)` Method sinkCleanup is used to destroy resources created by `Scheme.sinkPrepare(cascading.flow.FlowProcess, SinkCall)`.
`void`	`sinkConfInit(FlowProcess<Properties> flowProcess, Tap<Properties,InputStream,OutputStream> tap, Properties conf)` Method sinkInit initializes this instance as a sink.
`void`	`sinkPrepare(FlowProcess<Properties> flowProcess, SinkCall<PrintWriter,OutputStream> sinkCall)` Method sinkPrepare is used to initialize resources needed during each call of `Scheme.sink(cascading.flow.FlowProcess, SinkCall)`.
`boolean`	`source(FlowProcess<Properties> flowProcess, SourceCall<LineNumberReader,InputStream> sourceCall)` Method source will read a new "record" or value from `SourceCall.getInput()` and populate the available `Tuple` via `SourceCall.getIncomingEntry()` and return `true` on success or `false` if no more values available.
`void`	`sourceCleanup(FlowProcess<Properties> flowProcess, SourceCall<LineNumberReader,InputStream> sourceCall)` Method sourceCleanup is used to destroy resources created by `Scheme.sourcePrepare(cascading.flow.FlowProcess, SourceCall)`.
`void`	`sourceConfInit(FlowProcess<Properties> flowProcess, Tap<Properties,InputStream,OutputStream> tap, Properties conf)` Method sourceInit initializes this instance as a source.
`void`	`sourcePrepare(FlowProcess<Properties> flowProcess, SourceCall<LineNumberReader,InputStream> sourceCall)` Method sourcePrepare is used to initialize resources needed during each call of `Scheme.source(cascading.flow.FlowProcess, SourceCall)`.

Methods inherited from class cascading.scheme.Scheme
`equals, getNumSinkParts, getSinkFields, getSourceFields, getTrace, hashCode, isSink, isSource, presentSinkFieldsInternal, presentSourceFieldsInternal, retrieveSinkFields, setNumSinkParts, setSinkFields, setSourceFields, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Constructor Detail

TextDelimited

public TextDelimited()

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

TextDelimited

@ConstructorProperties(value={"hasHeader","delimiter"})
public TextDelimited(boolean hasHeader,
                                                String delimiter)

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

Parameters:: hasHeader -; delimiter -

TextDelimited

@ConstructorProperties(value={"hasHeader","delimiter","quote"})
public TextDelimited(boolean hasHeader,
                                                String delimiter,
                                                String quote)

Constructor TextDelimited creates a new TextDelimited instance sourcing Fields.UNKNOWN, sinking Fields.ALL and using TAB as the default delimiter.

Use this constructor if the source and sink fields will be resolved during planning, for example, when using with a Checkpoint Tap.

Parameters:: hasHeader -; delimiter -; quote -

TextDelimited

@ConstructorProperties(value="fields")
public TextDelimited(Fields fields)

Constructor TextDelimited creates a new TextDelimited instance with TAB as the default delimiter.

Parameters:: fields - of type Fields