Machine learning models for parsing remote sensing data have a wide range of societally relevant applications, but labels used to train these models can be difficult or impossible to acquire. This challenge has spurred research into self-supervised learning for remote sensing data aiming to unlock the use of machine learning in geographies or application domains where labelled datasets are small. Current self-supervised learning approaches for remote sensing data draw significant inspiration from techniques applied to natural images. However, remote sensing data has important differences from natural images -- for example, the temporal dimension is critical for many tasks and data is collected from many complementary sensors. We show we can create significantly smaller performant models by designing architectures and self-supervised training techniques specifically for remote sensing data. We introduce the Pretrained Remote Sensing Transformer (Presto), a transformer-based model pre-trained on remote sensing pixel-timeseries data. Presto excels at a wide variety of globally distributed remote sensing tasks and performs competitively with much larger models while requiring far less compute. Presto can be used for transfer learning or as a feature extractor for simple models, enabling efficient deployment at scale.

翻译：用于解析遥感数据的机器学习模型具有广泛的社会应用价值，但训练这些模型所需的数据标注往往难以获取或无法获得。这一挑战推动了遥感数据自监督学习研究的发展，旨在在标注数据集较小的地理区域或应用领域中实现机器学习的应用。当前遥感数据的自监督学习方法大量借鉴了自然图像处理技术。然而，遥感数据与自然图像存在显著差异——例如，时间维度对许多任务至关重要，且数据来自多种互补传感器。我们通过针对遥感数据设计专用架构和自监督训练技术，成功构建了体积显著缩小的高性能模型。本文提出预训练遥感变压器（Presto），这是一种基于遥感像素时间序列数据预训练的Transformer模型。Presto在多种全球分布的遥感任务中表现优异，与体积更大的模型相比具有同等竞争力，同时计算需求大幅降低。该模型既可用于迁移学习，也可作为简单模型的特征提取器，支持高效的大规模部署。