GLEAN: Generalized Deduplication Enabled Approximate Edge Analytics

The Internet of Things (IoT) has brought about exponential growth in sensor data. This has led to increasing demands for efficient and novel data transmission, storage and analytics solutions for sustainable IoT ecosystems. It has been shown that the Generalized Deduplication (GD) compression algorithm offers not only competitive compression ratio and throughput, but also random access properties that enable direct analytics of compressed data. In this paper, we thoroughly stresstest existing methods for direct analytics of GD compressed data with a diverse collection of 103 datasets, identify the need to optimise GD for analytics and develop a new version of GD to this end. We also propose the Generalized Deduplication Enabled Approximate Edge Analytics (GLEAN) framework. This framework applies the aforementioned analytics techniques at the Edge server to deliver end-to-end lossless data compression and highquality Edge analytics in the IoT, thereby addressing challenges related to data transmission, storage and analytics. Impressive analytics performance was achieved using this framework, with a median increase in k-means clustering error of just 2% relative to analytics performed on uncompressed data, while running 7.5x faster and requiring 3.9x less storage at the Edge server compared to universal compressors.