Large, pretrained models are commonly finetuned with imagery that is heavily augmented to mimic different conditions and scales, with the resulting models used for various tasks with imagery from a range of spatial scales. Such models overlook scale-specific information in the data for scale-dependent domains, such as remote sensing. In this paper, we present Scale-MAE, a pretraining method that explicitly learns relationships between data at different, known scales throughout the pretraining process. Scale-MAE pretrains a network by masking an input image at a known input scale, where the area of the Earth covered by the image determines the scale of the ViT positional encoding, not the image resolution. Scale-MAE encodes the masked image with a standard ViT backbone, and then decodes the masked image through a bandpass filter to reconstruct low/high frequency images at lower/higher scales. We find that tasking the network with reconstructing both low/high frequency images leads to robust multiscale representations for remote sensing imagery. Scale-MAE achieves an average of a $2.4 - 5.6\%$ non-parametric kNN classification improvement across eight remote sensing datasets compared to current state-of-the-art and obtains a $0.9$ mIoU to $1.7$ mIoU improvement on the SpaceNet building segmentation transfer task for a range of evaluation scales.

翻译：大型预训练模型通常通过对经过强烈增强以模拟不同条件和尺度的图像进行微调，从而用于处理来自不同空间尺度范围的各类图像任务。然而，此类模型在处理遥感等尺度依赖性领域时，会忽略数据中与尺度相关的特定信息。本文提出Scale-MAE，一种在预训练过程中明确学习不同已知尺度数据间关系的预训练方法。Scale-MAE通过以已知输入尺度对图像进行掩码预训练网络，其中图像覆盖的地球区域面积决定了ViT位置编码的尺度，而非图像分辨率。该方法采用标准ViT主干网络编码掩码图像，随后通过带通滤波器对掩码图像进行解码，以重构低/高频下的高/低频图像。研究发现，让网络同时重构低/高频图像能够为遥感影像生成稳健的多尺度表示。与当前最先进方法相比，Scale-MAE在八个遥感数据集上实现了平均2.4%-5.6%的非参数kNN分类性能提升，并在多种评估尺度下的SpaceNet建筑物分割迁移任务中获得了0.9至1.7 mIoU的改进。