org.apache.hadoop.chukwa.extraction.archive

Class SinkArchiver

java.lang.Object

org.apache.hadoop.chukwa.extraction.archive.SinkArchiver

All Implemented Interfaces:

CHUKWA_CONSTANT

public class SinkArchiver
extends Object
implements CHUKWA_CONSTANT

A lightweight tool for archiving, suitable for small-to-medium-size Chukwa deployments that don't use Demux. Grabs everything in the data sink, runs the Archiver MapReduce job, then promotes output to the archive dir. Input is determined by conf option chukwaArchiveDir; defaults to /chukwa/logs Uses /chukwa/archivesProcessing/mr[Input/Output] as tmp storage Outputs to /chukwa/archives

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	DATA_SINK_FILTER

Fields inherited from interface org.apache.hadoop.chukwa.extraction.CHUKWA_CONSTANT

ARCHIVES_IN_ERROR_DIR_NAME, ARCHIVES_MR_INPUT_DIR_NAME, ARCHIVES_MR_OUTPUT_DIR_NAME, ARCHIVES_PROCESSING_DIR_NAME, CHUKWA_ARCHIVE_DIR_FIELD, CHUKWA_ARCHIVE_MAX_ERROR_COUNT_FIELD, CHUKWA_DATA_SINK_DIR_FIELD, CHUKWA_DEMUX_MAX_ERROR_COUNT_FIELD, CHUKWA_DEMUX_REDUCER_COUNT_FIELD, CHUKWA_NAGIOS_HOST_FIELD, CHUKWA_NAGIOS_PORT_FIELD, CHUKWA_POST_PROCESS_DIR_FIELD, CHUKWA_POSTPROCESS_IN_ERROR_DIR_FIELD, CHUKWA_POSTPROCESS_MAX_ERROR_COUNT_FIELD, CHUKWA_REPORTING_HOST_FIELD, CHUKWA_ROOT_DIR_FIELD, CHUKWA_ROOT_REPOS_DIR_FIELD, DEFAULT_CHUKWA_DATASINK_DIR_NAME, DEFAULT_CHUKWA_LOGS_DIR_NAME, DEFAULT_CHUKWA_POSTPROCESS_DIR_NAME, DEFAULT_CHUKWA_ROOT_DIR_NAME, DEFAULT_DEMUX_IN_ERROR_DIR_NAME, DEFAULT_DEMUX_MR_INPUT_DIR_NAME, DEFAULT_DEMUX_MR_OUTPUT_DIR_NAME, DEFAULT_DEMUX_PROCESSING_DIR_NAME, DEFAULT_FINAL_ARCHIVES, DEFAULT_POSTPROCESS_IN_ERROR_DIR_NAME, DEFAULT_REPOS_DIR_NAME, HDFS_DEFAULT_NAME_FIELD, HIERARCHY_CONNECTOR, INCLUDE_KEY_IN_PARTITIONER, POST_DEMUX_DATA_LOADER, POST_DEMUX_SUCCESS_ACTION, WRITER_HDFS_FILESYSTEM_FIELD

Constructor Summary

Constructors

Constructor and Description
SinkArchiver()

Method Summary

Modifier and Type	Method and Description
void	exec(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
static void	main(String[] args)
void	promoteAndMerge(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dest) Merges the contents of src into dest.
int	runMapRedJob(org.apache.hadoop.conf.Configuration conf, String in, String out)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DATA_SINK_FILTER

public static final org.apache.hadoop.fs.PathFilter DATA_SINK_FILTER

Constructor Detail

SinkArchiver

public SinkArchiver()

Method Detail

main

public static void main(String[] args)

exec

public void exec(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.conf.Configuration conf)

runMapRedJob

public int runMapRedJob(org.apache.hadoop.conf.Configuration conf,
                        String in,
                        String out)
                 throws Exception

Throws:

Exception

promoteAndMerge

public void promoteAndMerge(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path src,
                            org.apache.hadoop.fs.Path dest)
                     throws IOException

Merges the contents of src into dest. If src is a file, moves it to dest.

Parameters:

fs - the filesystem in question

src - a file or directory to merge into dest

dest - a directory to merge into

Throws:

IOException - if error in promote and merge