Labeling Scheduler: A Flexible Labeling-Based Jointly Scheduling Approach for Big Data Analysis

The emerging Non-Volatile Memory (NVM) technology has given rise to an opportunity to accelerate big data analysis. In this paper, we investigate the joint job and data scheduling problem in private cloud data center with a hybrid storage system, and we propose Labeling Scheduler, a flexible labeling-based approach for jointly scheduling. The core idea of the approach is to introduce the labeling system to characterize the features of big data analysis jobs and data objects, and conduct data replacement dynamically between NVM and disk. To the best of our knowledge, this is the first work to introduce the labeling methodology to the big data analysis problem in the cloud data center with a hybrid storage system. We conduct extensive simulations and the simulation results show that the Labeling Scheduler has a significant improvement on system utility compared to the method without labeling information. In addition, the Labeling Scheduler guarantees a high NVM hit rate, which is valuable for NVM endurance enhancement.