Geospatial technologies are becoming increasingly essential in our world for a large range of tasks, such as earth monitoring and natural disaster response. To help improve the applicability and performance of deep learning models on these geospatial tasks, various works have pursued the idea of a geospatial foundation model, i.e., training networks from scratch on a large corpus of remote sensing imagery. However, this approach often requires a significant amount of data and training time to achieve suitable performance, especially when employing large state-of-the-art transformer models. In light of these challenges, we investigate a sustainable approach to building geospatial foundation models. In our investigations, we discover two important factors in the process. First, we find that the selection of pretraining data matters, even within the geospatial domain. We therefore gather a concise yet effective dataset for pretraining. Second, we find that available pretrained models on diverse datasets like ImageNet-22k should not be ignored when building geospatial foundation models, as their representations are still surprisingly effective. Rather, by leveraging their representations, we can build strong models for geospatial applications in a sustainable manner. To this end, we formulate a multi-objective continual pretraining approach for training sustainable geospatial foundation models. We experiment on a wide variety of downstream datasets and tasks, achieving strong performance across the board in comparison to ImageNet baselines and state-of-the-art geospatial pretrained models.

翻译：摘要：地理空间技术在地球监测和自然灾害响应等广泛任务中日益重要。为提升深度学习模型在这些地理空间任务中的适用性与性能，多项研究致力于构建地理空间基础模型，即从零开始在大量遥感图像语料上训练网络。然而，该方法通常需要大量数据和训练时间才能达到理想性能，尤其在采用先进的大型Transformer模型时。针对这些挑战，我们探索了一种可持续构建地理空间基础模型的方法。研究中发现两个关键因素：首先，预训练数据的选择至关重要，即使在地理空间领域内也是如此。因此，我们收集了一个简洁而有效的预训练数据集。其次，我们发现ImageNet-22k等多样化数据集上的现有预训练模型在地理空间基础模型构建中不应被忽视，其表征能力仍出奇有效。通过利用这些表征，我们可以可持续地构建适用于地理空间应用的强模型。为此，我们提出了一种多目标持续预训练方法，用于训练可持续的地理空间基础模型。我们在多种下游数据集和任务上进行实验，结果表明，与ImageNet基线及最先进的地理空间预训练模型相比，本方法在各项任务中均取得了优异性能。