Deep learning has largely reshaped remote sensing (RS) research for aerial image understanding and made a great success. Nevertheless, most of the existing deep models are initialized with the ImageNet pretrained weights. Since natural images inevitably present a large domain gap relative to aerial images, probably limiting the finetuning performance on downstream aerial scene tasks. This issue motivates us to conduct an empirical study of remote sensing pretraining (RSP) on aerial images. To this end, we train different networks from scratch with the help of the largest RS scene recognition dataset up to now -- MillionAID, to obtain a series of RS pretrained backbones, including both convolutional neural networks (CNN) and vision transformers such as Swin and ViTAE, which have shown promising performance on computer vision tasks. Then, we investigate the impact of RSP on representative downstream tasks including scene recognition, semantic segmentation, object detection, and change detection using these CNN and vision transformer backbones. Empirical study shows that RSP can help deliver distinctive performances in scene recognition tasks and in perceiving RS related semantics such as "Bridge" and "Airplane". We also find that, although RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, it may still suffer from task discrepancies, where downstream tasks require different representations from scene recognition tasks. These findings call for further research efforts on both large-scale pretraining datasets and effective pretraining methods. The codes and pretrained models will be released at https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.

翻译：深度学习极大地重塑了遥感领域中对航空影像理解的研究，并取得了巨大成功。然而，大多数现有深度模型均以ImageNet预训练权重进行初始化。由于自然图像与航空图像间不可避免存在显著的领域差距，这可能会限制下游航空场景任务的微调性能。这一发现促使我们针对航空影像开展遥感预训练的实证研究。为此，我们借助迄今最大的遥感场景识别数据集——MillionAID，从零开始训练不同网络，获得了一系列遥感预训练主干网络，包括卷积神经网络以及视觉Transformer（如Swin和ViTAE），这些模型已在计算机视觉任务中展现出优异性能。随后，我们利用这些CNN和视觉Transformer主干网络，探究了遥感预训练对代表性下游任务（包括场景识别、语义分割、目标检测和变化检测）的影响。实证研究表明，遥感预训练能够显著提升场景识别任务的表现，并增强对"桥梁"、"飞机"等遥感相关语义的感知能力。我们还发现，尽管遥感预训练缓解了传统ImageNet预训练在遥感图像上的数据差异问题，但仍可能面临任务差异挑战——下游任务所需表征与场景识别任务存在差异。这些发现为大规模预训练数据集与高效预训练方法的研究指明了方向。相关代码与预训练模型将发布于https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing。