org.apache.hadoop.chukwa.datacollection.adaptor.filetailer

Class LWFTAdaptor

java.lang.Object

org.apache.hadoop.chukwa.datacollection.adaptor.AbstractAdaptor

org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.LWFTAdaptor

All Implemented Interfaces:

Adaptor

Direct Known Subclasses:

FileTailingAdaptor

public class LWFTAdaptor
extends AbstractAdaptor

A base class for file tailing adaptors. Intended to mandate as little policy as possible, and to use as few system resources as possible. If the file does not exist, this class will continue to retry quietly forever and will start tailing if it's eventually created.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
static int	DEFAULT_MAX_READ_SIZE This is the maximum amount we'll read from any one file before moving on to the next.
protected long	fileReadOffset next PHYSICAL offset to read
static int	MAX_READ_SIZE
static String	MAX_READ_SIZE_OPT
protected long	offsetOfFirstByte The logical offset of the first byte of the file
protected static org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.FileTailer	tailer

Fields inherited from class org.apache.hadoop.chukwa.datacollection.adaptor.AbstractAdaptor

adaptorID, control, dest, type

Constructor Summary

Constructors

Constructor and Description
LWFTAdaptor()

Method Summary

Methods

Modifier and Type	Method and Description
protected int	extractRecords(ChunkReceiver eq, long buffOffsetInFile, byte[] buf) Extract records from a byte sequence
String	getCurrentStatus() Return the adaptor's state Should not include class name or byte offset, which are written by caller.
String	getStreamName()
void	hardStop() Signals this adaptor to come to an abrupt stop, as quickly as it can.
String	parseArgs(String params) Parse args, return stream name.
long	shutdown() Signals this adaptor to come to an orderly stop.
long	shutdown(AdaptorShutdownPolicy shutdownPolicy) Signals this adaptor to come to an orderly stop.
protected boolean	slurp(long len, RandomAccessFile reader)
void	start(long offset)
boolean	tailFile(ChunkReceiver eq)
String	toString()

Methods inherited from class org.apache.hadoop.chukwa.datacollection.adaptor.AbstractAdaptor

deregisterAndStop, getType, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_MAX_READ_SIZE

public static final int DEFAULT_MAX_READ_SIZE

This is the maximum amount we'll read from any one file before moving on to the next. This way, we get quick response time for other files if one file is growing rapidly.

See Also:

Constant Field Values

MAX_READ_SIZE_OPT

public static final String MAX_READ_SIZE_OPT

See Also:

Constant Field Values

MAX_READ_SIZE

public static int MAX_READ_SIZE

tailer

protected static org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.FileTailer tailer

fileReadOffset

protected long fileReadOffset

next PHYSICAL offset to read

offsetOfFirstByte

protected long offsetOfFirstByte

The logical offset of the first byte of the file

conf

protected org.apache.hadoop.conf.Configuration conf

Constructor Detail

LWFTAdaptor

public LWFTAdaptor()

Method Detail

start

public void start(long offset)

Specified by:

start in class AbstractAdaptor

getCurrentStatus

public String getCurrentStatus()

Description copied from interface: Adaptor

Return the adaptor's state Should not include class name or byte offset, which are written by caller. The datatype should, however, be written by this method.

Returns:

the adaptor state as a string

See Also:

Adaptor.getCurrentStatus()

toString

public String toString()

Overrides:

toString in class Object

getStreamName

public String getStreamName()

parseArgs

public String parseArgs(String params)

Description copied from interface: Adaptor

Parse args, return stream name. Do not start running. Return the stream name, given params. The stream name is the part of the Adaptor status that's used to determine uniqueness.

Returns:

Stream name as a string, null if params are malformed

hardStop

public void hardStop()
              throws AdaptorException

Description copied from interface: Adaptor

Signals this adaptor to come to an abrupt stop, as quickly as it can. The use case here is "Whups, I didn't mean to start that adaptor tailing a gigabyte file, stop it now". Adaptors might need to do something nontrivial here, e.g., in the case in which they have registered periodic timer interrupts, or use a shared worker thread from which they need to disengage. This method is synchronous: In other words, after shutdown() returns, no new data should be written.

Throws:

AdaptorException

shutdown

public long shutdown()
              throws AdaptorException

Description copied from interface: Adaptor

Signals this adaptor to come to an orderly stop. The adaptor ought to push out all the data it can before exiting. This method is synchronous up to 60 seconds

Returns:

the logical offset at which the adaptor stops

Throws:

AdaptorException

shutdown

public long shutdown(AdaptorShutdownPolicy shutdownPolicy)
              throws AdaptorException

Description copied from interface: Adaptor

Signals this adaptor to come to an orderly stop. The adaptor ought to push out all the data it can before exiting depending of the shutdown policy

Returns:

the logical offset at which the adaptor was when the method return

Throws:

AdaptorException

extractRecords

protected int extractRecords(ChunkReceiver eq,
                 long buffOffsetInFile,
                 byte[] buf)
                      throws InterruptedException

Extract records from a byte sequence

Parameters:

eq - the queue to stick the new chunk[s] in

buffOffsetInFile - the byte offset in the stream at which buf[] begins

buf - the byte buffer to extract records from

Returns:

the number of bytes processed

Throws:

InterruptedException

slurp

protected boolean slurp(long len,
            RandomAccessFile reader)
                 throws IOException,
                        InterruptedException

Throws:

IOException

InterruptedException

tailFile

public boolean tailFile(ChunkReceiver eq)
                 throws InterruptedException

Throws:

InterruptedException