Self-Supervised Cross-Sensor Spectral Image Fusion: A cross-domain multimodal reciprocal learning framework

Spectral images provide richer information than RGB images, yet face a trade-off between spatial and spectral resolution: multispectral images (MSIs) offer high spatial details, whereas hyperspectral images (HSIs) capture fine spectral channels. Although spectral image fusion theoretically leverages the complementary strengths of HSIs and MSIs, existing methods often suffer from spectral distortion in the reconstructed HSI. This issue mainly arises because HSI and MSI sensors capture electromagnetic radiation differently, resulting in inconsistencies in the information distribution of the same scene. To alleviate these issues, this paper proposes a Cross-Domain Multimodal Reciprocal Learning (CMRL) framework for self-supervised spectral image fusion. CMRL enables self-supervised interactions across the frequency, spatial, and spectral domains via a contrastive learning strategy, maximizing the utilization of crucial information from each modality during multi-sensor spectral fusion. Furthermore, CMRL can leverage the global context-aware representations provided by vision Transformer (ViT) and vision Mamba (ViM) architectures to further explore the homogeneity between modalities. This design facilitates generalization across datasets acquired on different platforms. Based on this framework, we develop three variants: Independent CMRL (CMRL-I), ViT-based CMRL (CMRL-T), and ViM-based CMRL (CMRL-M). The code and new dataset are released at https://github.com/yuanchaosu/GRSM-CMRL