EaCRS: an extendible array based compression scheme for high dimensional data

Multidimensional arrays are becoming important data structure for handling large scale multidimensional data; e.g., in scientific databases or MOLAP databases. Due to the increasing size of the data warehouses and high degree of sparsity, it becomes necessity to develop a suitable scheme to compress the multidimensional array in an efficient way so that it takes comparatively low memory storage. In this paper, we propose a new compression scheme namely extendible array based Compressed Row Storage (EaCRS), for large multidimensional sparse array. The main idea of this scheme is to compress the subarrays found from the existing extendible array using CRS method. To evaluate the proposed scheme, we compare it to the CRS on Traditional multidimensional array (TMA). Both analytical analysis and experimental test were conducted. In the analytical analysis, we analyze the CRS and EaCRS schemes in terms of the space requirement and the maximum range of usable data density for practical applications. The analytical analysis and experimental results show that the EaCRS scheme is superior to the CRS scheme for all the evaluated criteria.