Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to inject spatial details from panchromatic images into multispectral images (MS) to obtain high-resolution multispectral images. Since deep learning has received widespread attention because of its powerful fitting ability and efficient feature extraction, a variety of pan-sharpening methods have been proposed to achieve remarkable performance. However, current pan-sharpening methods usually require the paired panchromatic (PAN) and MS images as input, which limits their usage in some scenarios. To address this issue, in this paper we observe that the spatial details from PAN images are mainly high-frequency cues, i.e., the edges reflect the contour of input PAN images. This motivates us to develop a PAN-agnostic representation to store some base edges, so as to compose the contour for the corresponding PAN image via them. As a result, we can perform the pan-sharpening task with only the MS image when inference. To this end, a memory-based network is adapted to extract and memorize the spatial details during the training phase and is used to replace the process of obtaining spatial information from PAN images when inference, which is called Memory-based Spatial Details Network (MSDN). Finally, we integrate the proposed MSDN module into the existing deep learning-based pan-sharpening methods to achieve an end-to-end pan-sharpening network. With extensive experiments on the Gaofen1 and WorldView-4 satellites, we verify that our method constructs good spatial details without PAN images and achieves the best performance. The code is available at https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git.

翻译：全色锐化作为遥感系统中最常用的技术之一，旨在将全色图像的空间细节注入多光谱图像（MS），以获得高分辨率多光谱图像。由于深度学习因其强大的拟合能力和高效的特征提取而受到广泛关注，目前已提出多种全色锐化方法并取得了显著性能。然而，当前的全色锐化方法通常需要成对的全色（PAN）和MS图像作为输入，这限制了其在某些场景下的应用。针对这一问题，本文观察到PAN图像中的空间细节主要是高频线索，即边缘反映了输入PAN图像的轮廓。这启发我们开发一种PAN无关表示来存储一些基础边缘，从而通过它们合成对应PAN图像的轮廓。因此，我们可以在推理时仅使用MS图像执行全色锐化任务。为此，我们设计了一种基于记忆的网络，在训练阶段提取并记忆空间细节，并在推理时替代从PAN图像获取空间信息的过程，称为基于记忆的空间细节网络（MSDN）。最后，我们将所提出的MSDN模块集成到现有基于深度学习的全色锐化方法中，实现端到端的全色锐化网络。通过在高分一号和WorldView-4卫星上的大量实验，我们验证了该方法无需PAN图像即可构建良好的空间细节，并达到了最优性能。代码可在https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git获取。