Machine learning algorithms for parsing remote sensing data have a wide range of societally relevant applications, but labels used to train these algorithms can be difficult or impossible to acquire. This challenge has spurred research into self-supervised learning for remote sensing data aiming to unlock the use of machine learning in geographies or application domains where labelled datasets are small. Current self-supervised learning approaches for remote sensing data draw significant inspiration from techniques applied to natural images. However, remote sensing data has important differences from natural images -- for example, the temporal dimension is critical for many tasks and data is collected from many complementary sensors. We show that designing models and self-supervised training techniques specifically for remote sensing data results in both smaller and more performant models. We introduce the Pretrained Remote Sensing Transformer (Presto), a transformer-based model pre-trained on remote sensing pixel-timeseries data. Presto excels at a wide variety of globally distributed remote sensing tasks and outperforms much larger models. Presto can be used for transfer learning or as a feature extractor for simple models, enabling efficient deployment at scale.

翻译：用于解析遥感数据的机器学习算法具有广泛的社会相关应用，但训练这些算法所需的标签数据往往难以甚至无法获取。这一挑战催生了针对遥感数据的自监督学习研究，旨在将机器学习应用于标注数据集较小的地理区域或应用领域。当前遥感数据的自监督学习方法多受自然图像处理技术的启发，然而遥感数据与自然图像存在重要差异——例如时间维度对许多任务至关重要，且数据来自多种互补传感器。研究表明，针对遥感数据特性设计的模型与自监督训练技术，能够同时实现更小体积与更优性能的模型。我们提出预训练遥感Transformer（Presto），这是一种基于Transformer架构、在遥感像素时间序列数据上预训练的模型。Presto在多种全球分布的遥感任务中表现优异，甚至超越更大规模的模型。该模型既可用于迁移学习，亦可作为简单模型的特征提取器，从而支持大规模高效部署。