Self-supervised frameworks for representation learning have recently stirred up interest among the remote sensing community, given their potential to mitigate the high labeling costs associated with curating large satellite image datasets. In the realm of multimodal data fusion, while the often used contrastive learning methods can help bridging the domain gap between different sensor types, they rely on data augmentations techniques that require expertise and careful design, especially for multispectral remote sensing data. A possible but rather scarcely studied way to circumvent these limitations is to use a masked image modelling based pretraining strategy. In this paper, we introduce Fus-MAE, a self-supervised learning framework based on masked autoencoders that uses cross-attention to perform early and feature-level data fusion between synthetic aperture radar and multispectral optical data - two modalities with a significant domain gap. Our empirical findings demonstrate that Fus-MAE can effectively compete with contrastive learning strategies tailored for SAR-optical data fusion and outperforms other masked-autoencoders frameworks trained on a larger corpus.

翻译：自监督表征学习框架因其在缓解大规模卫星图像数据集标注成本高昂问题方面的潜力，近期引起了遥感领域的广泛关注。在多模态数据融合中，尽管常用的对比学习方法有助于弥合不同传感器类型之间的领域差异，但其依赖的数据增强技术需要专业知识与精心设计——尤其针对多光谱遥感数据。一种可能但鲜有研究的解决方案是采用基于掩码图像建模的预训练策略。本文提出Fus-MAE，一种基于掩码自编码器的自监督学习框架，该框架通过交叉注意力在合成孔径雷达与多光谱光学数据（两者存在显著领域差异）之间实现早期特征级数据融合。实验结果表明，Fus-MAE能够有效媲美专门针对SAR-光学数据融合设计的对比学习策略，并优于基于更大语料库训练的其他掩码自编码器框架。