Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to inject spatial details from panchromatic images into multispectral images (MS) to obtain high-resolution multispectral images. Since deep learning has received widespread attention because of its powerful fitting ability and efficient feature extraction, a variety of pan-sharpening methods have been proposed to achieve remarkable performance. However, current pan-sharpening methods usually require the paired panchromatic (PAN) and MS images as input, which limits their usage in some scenarios. To address this issue, in this paper we observe that the spatial details from PAN images are mainly high-frequency cues, i.e., the edges reflect the contour of input PAN images. This motivates us to develop a PAN-agnostic representation to store some base edges, so as to compose the contour for the corresponding PAN image via them. As a result, we can perform the pan-sharpening task with only the MS image when inference. To this end, a memory-based network is adapted to extract and memorize the spatial details during the training phase and is used to replace the process of obtaining spatial information from PAN images when inference, which is called Memory-based Spatial Details Network (MSDN). Finally, we integrate the proposed MSDN module into the existing deep learning-based pan-sharpening methods to achieve an end-to-end pan-sharpening network. With extensive experiments on the Gaofen1 and WorldView-4 satellites, we verify that our method constructs good spatial details without PAN images and achieves the best performance. The code is available at https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git.

翻译：全色锐化作为遥感系统中最常用的技术之一，旨在将全色图像中的空间细节注入多光谱图像，以获取高分辨率多光谱图像。由于深度学习因其强大的拟合能力和高效的特征提取而受到广泛关注，研究者已提出多种全色锐化方法并取得了显著性能。然而，当前全色锐化方法通常需要成对的全色和多光谱图像作为输入，这限制了其在某些场景中的应用。为解决该问题，本文观察到全色图像中的空间细节主要为高频线索，即边缘反映了输入全色图像的轮廓。这一发现启发我们开发一种与全色无关的表示来存储基础边缘，从而通过它们组合对应全色图像的轮廓。由此，我们可在推理时仅使用多光谱图像执行全色锐化任务。为此，我们设计了基于记忆的网络，在训练阶段提取并记忆空间细节，并在推理时替代从全色图像获取空间信息的过程，该网络被称为基于记忆的空间细节网络(MSDN)。最后，我们将所提出的MSDN模块集成到现有基于深度学习的全色锐化方法中，构建端到端的全色锐化网络。通过在高分一号和WorldView-4卫星上的大量实验，我们验证了该方法无需全色图像即可构建良好的空间细节，并取得了最优性能。代码开源地址：https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git。