org.apache.hadoop.chukwa.util

Class HierarchyDataType

java.lang.Object

org.apache.hadoop.chukwa.util.HierarchyDataType

public class HierarchyDataType
extends Object

To support hierarchyDataType according to CHUKWA-648, which is quite similar to the idea of Hive's Partition. For example, the user can define the dataType as "datatypeLevel1/dataTypeLevel2/dataTypeLevel3" instead of a flat structure like: "datatypeLevel1_datatTypeLeve2_dataTypeLevel3"

The hierarchyDataType makes the filtering work much more easy when doing the analysis job. For example, if the user focuses on all data under "datatypeLevel1/dataTypeLevel2" category, he only needs to go through all level2 related sub-directories.

Constructor Summary

Constructors

Constructor and Description
HierarchyDataType()

Method Summary

Modifier and Type	Method and Description
static String	getDataType(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.Path cluster) Get the hierarchyDataType format from the directory.
static String	getHierarchyDataTypeDirectory(String datasource) Transform the hierarchyDataType filename into its directory name (with slash mark)
static String	getHierarchyDataTypeFileName(String datasource) Transform the hierarchyDatatType directory into its filename (without any slash mark)
static List<org.apache.hadoop.fs.FileStatus>	globStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) List all files under certain path and its sub-directories
static List<org.apache.hadoop.fs.FileStatus>	globStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter filter, boolean recursive) List all matched files under the directory and its sub-dirs
static String	trimSlash(String datasource) Get the directory without first and last slash mark.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HierarchyDataType

public HierarchyDataType()

Method Detail

globStatus

public static List<org.apache.hadoop.fs.FileStatus> globStatus(org.apache.hadoop.fs.FileSystem fs,
                                                               org.apache.hadoop.fs.Path path,
                                                               org.apache.hadoop.fs.PathFilter filter,
                                                               boolean recursive)

List all matched files under the directory and its sub-dirs

Parameters:

fs - The file system

path - The parent folder

filter - The pattern matcher to filter the required files

recursive -

Returns:

List of file status

globStatus

public static List<org.apache.hadoop.fs.FileStatus> globStatus(org.apache.hadoop.fs.FileSystem fs,
                                                               org.apache.hadoop.fs.Path path,
                                                               boolean recursive)

List all files under certain path and its sub-directories

Parameters:

fs - The file system

path - The parent folder

recursive -

Returns:

The list of all sub-dirs

getDataType

public static String getDataType(org.apache.hadoop.fs.Path path,
                                 org.apache.hadoop.fs.Path cluster)

Get the hierarchyDataType format from the directory.

Parameters:

path - The data path

cluster - The cluster's folder

Returns:

The hierarchyDataType

trimSlash

public static String trimSlash(String datasource)

Get the directory without first and last slash mark.

Parameters:

datasource -

Returns:

directory name

getHierarchyDataTypeFileName

public static String getHierarchyDataTypeFileName(String datasource)

Transform the hierarchyDatatType directory into its filename (without any slash mark)

Parameters:

datasource -

Returns:

filename

getHierarchyDataTypeDirectory

public static String getHierarchyDataTypeDirectory(String datasource)

Transform the hierarchyDataType filename into its directory name (with slash mark)

Parameters:

datasource -

Returns:

directory name