An In-Memory Data-Cube Aware Distributed Data Discovery Across Clouds for Remote Sensing Big Data.

With the booming of high-resolution Earth observation and open-data efforts, petabyte-scale Earth observation data have been available for free access. Due to the unprecedented availability of big data deluge, regional to global spatio-temporal analysis has been significantly challenged with the huge computational barriers, the tedious cycles of "download-preprocess-store-analyze" leading to excessive data downloading overhead, and the acquisition-oriented 2-D file-based structure, which is not fit for spatio-temporal analysis. The Earth observation data cube (EODC) paradigm revolutionizes the traditional way of storing, managing, and analyzing spatio-temporal RS data, and solves problems of easy-to-use of RS data to a certain extent. However, different EODC solutions are becoming "information silos." Therefore, the sharing and joint use of remote sensing (RS) data across EODCs have become extremely challenging. To address the abovementioned challenges, we proposed a method of in-memory distributed data cube autodiscovery and retrieval across clouds. We construct a distributed in-memory data orchestration across clouds to shield the heterogeneity of the EODC storage solutions, solving "information silos" problems, and we put forward a larger-sites-first and spatio-temporal aware RS data discovery strategy, which can automatically discover data across clouds for requirements. Based on the data cube paradigm, this article proposes a quality-first data filtering strategy, which can help users to filter out high-quality data covering the target spatio-temporal range from the huge amount of data, and solve the problem of data cube joint retrieval and efficient use across clouds. In addition, we have confirmed that our method is effective and efficient through comparative experiments.