8.10 Maintaining State in Operations

When creating custom Operations (Function, Filter,Aggregator, or Buffer) do not store operation state in class fields. For example, if implementing a custom 'counter' Aggregator, do not create a field named 'count' and increment it on every Aggregator.aggregate() call. There is no guarantee your Operation will be called from a single thread in a JVM, future version of Hadoop could execute the same operation from multiple threads.

To maintain state across Operation calls, create and initialize a "context" object that is maintained by the appropriate OperationCall (FilterCall,FunctionCall, AggregatorCall, and BufferCall). In the example above, store an Integer 0 in the AggregatorCall passed to the Aggregator.start() method and increment it in the Aggregator.aggregate() method.

Copyright © 2007-2008 Concurrent, Inc. All Rights Reserved.