Pixel-to-Abundance Translation: Conditional Generative Adversarial Networks Based on Patch Transformer for Hyperspectral Unmixing

Spectral unmixing is a significant challenge in hyperspectral image processing. Existing unmixing methods utilize prior knowledge about the abundance distribution to solve the regularization optimization problem, where the difficulty lies in choosing appropriate prior knowledge and solving the complex regularization optimization problem. To solve these problems, we propose a hyperspectral conditional generative adversarial network (HyperGAN) method as a generic unmixing framework based on the following assumption: The unmixing process from pixel to abundance can be regarded as a transformation of two modalities with an internal specific relationship. The proposed HyperGAN is composed of a generator and a discriminator, the former completes the modal conversion from mixed hyperspectral pixel patch to the abundance of corresponding endmember of the central pixel and the latter is used to distinguish whether the distribution and structure of generated abundance are the same as the true ones. In addition, we propose hyperspectral image (HSI) patch transformer as the main component of the generator, which utilizes adaptive attention score to capture the internal pixels correlation of the HSI patch and leverages the spatial-spectral information in a fine-grained way to achieve optimization of the unmixing process. Furthermore, we propose a data synthesis method based on superpixel segmentation and random split to get synthetic data with a spatial structure to further evaluate the effectiveness of our model for unmixing. Experiments on synthetic data and real hyperspectral data achieve impressive results compared with state-of-the-art competitors.