Machine learning algorithms for parsing remote sensing data have a wide range of societally relevant applications, but labels used to train these algorithms can be difficult or impossible to acquire. This challenge has spurred research into self-supervised learning for remote sensing data aiming to unlock the use of machine learning in geographies or application domains where labelled datasets are small. Current self-supervised learning approaches for remote sensing data draw significant inspiration from techniques applied to natural images. However, remote sensing data has important differences from natural images -- for example, the temporal dimension is critical for many tasks and data is collected from many complementary sensors. We show that designing models and self-supervised training techniques specifically for remote sensing data results in both smaller and more performant models. We introduce the Pretrained Remote Sensing Transformer (Presto), a transformer-based model pre-trained on remote sensing pixel-timeseries data. Presto excels at a wide variety of globally distributed remote sensing tasks and outperforms much larger models. Presto can be used for transfer learning or as a feature extractor for simple models, enabling efficient deployment at scale.

翻译：解析遥感数据的机器学习算法具有广泛的社会应用价值，但训练这些算法所需的标签往往难以甚至无法获取。这一挑战推动了面向遥感数据的自监督学习研究，旨在破解标注数据集稀缺的地理区域或应用领域中的机器学习应用难题。当前遥感数据的自监督学习方法主要借鉴自然图像处理技术，但遥感数据与自然图像存在本质差异——例如时间维度对许多任务至关重要，且数据来自多个互补传感器。研究表明，针对遥感数据特性设计的模型与自监督训练技术，既能获得更轻量的模型结构，又能提升模型性能。我们提出预训练遥感Transformer（Presto），这是一种基于Transformer架构、在遥感像素时间序列数据上预训练的模型。Presto在多种全球分布的遥感任务中表现卓越，性能超越规模更大的模型。该模型既可应用于迁移学习，也可作为简单模型的特征提取器，从而实现大规模高效部署。