New Checkpoint and Rollback for High Availability of Mapreduce Computing

Abstract

MapReduce is a programming model and an associated implementation for processing and generating large data sets, so called big data. A MapReduce job usually splits the input data-set into independent chunks which are processed by the maptasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. If an error occurs in a name node other name node will take over the failed node and continues its execution. Other than data node failure, if an error occurs during the program execution itself then there must be a detection and recovery steps to correct the error.A solution for this problem is to implement the checkpoint and rollback mechanism in the system. When memory error occurs in the MapReduce program then execution in all the data nodes will be stopped and it starts all over from the starting phase in hadoop. The proposed methodology is to detect the heap space error [10] and provide a recovery operations by employing a new checkpoint and recovery process. In order to realize this, a new phase based checkpoint and rollback is proposed versus the hadoop default configuration. Once an error occurs in hadoop, the memory size required by the program is raised then the configuration file setting is modified and then a checkpoint is set and from there next phases will be executed. In this way, the entire already completed phases are not needed to be re-executed. From the experimental results, the hadoop availability is increased to 53.22% compared to the default hadoop configuration thereby decreasing the running time of the application.