Fusion of a panchromatic (PAN) image and corresponding multispectral (MS) image is also known as pansharpening, which aims to combine abundant spatial details of PAN and spectral information of MS. Due to the absence of high-resolution MS images, available deep-learning-based methods usually follow the paradigm of training at reduced resolution and testing at both reduced and full resolution. When taking original MS and PAN images as inputs, they always obtain sub-optimal results due to the scale variation. In this paper, we propose to explore the self-supervised representation of pansharpening by designing a cross-predictive diffusion model, named CrossDiff. It has two-stage training. In the first stage, we introduce a cross-predictive pretext task to pre-train the UNet structure based on conditional DDPM, while in the second stage, the encoders of the UNets are frozen to directly extract spatial and spectral features from PAN and MS, and only the fusion head is trained to adapt for pansharpening task. Extensive experiments show the effectiveness and superiority of the proposed model compared with state-of-the-art supervised and unsupervised methods. Besides, the cross-sensor experiments also verify the generalization ability of proposed self-supervised representation learners for other satellite's datasets. We will release our code for reproducibility.

翻译：全色（PAN）图像与对应多光谱（MS）图像的融合被称为全色锐化，其目标在于融合PAN的丰富空间细节与MS的光谱信息。由于缺乏高分辨率MS图像，现有深度学习方法通常遵循低分辨率训练、低分辨率与全分辨率共同测试的范式。当以原始MS和PAN图像作为输入时，因尺度变化问题，此类方法往往仅能获得次优结果。本文提出通过设计名为CrossDiff的交叉预测扩散模型，探索全色锐化的自监督表示。该模型采用两阶段训练：第一阶段引入交叉预测预训练任务，基于条件DDPM对UNet结构进行预训练；第二阶段冻结UNet编码器以直接提取PAN和MS的空间与光谱特征，仅训练融合头以适应全色锐化任务。大量实验表明，与当前最先进的监督与无监督方法相比，所提模型具有有效性与优越性。此外，跨传感器实验验证了所提自监督表示学习器对其他卫星数据集具有泛化能力。为保障可复现性，我们将公开代码。