Self-supervised learning has emerged as a powerful paradigm for pretraining foundation models using large-scale data. Existing pretraining approaches predominantly rely on masked reconstruction or next-token prediction strategies, demonstrating strong performance across various downstream tasks, including geoscience applications. However, these approaches do not fully capture the knowledge of causal interplay between different geospatial and environmental variables. To address this limitation, we propose Knowledge Guided Variable-Step Forecasting (KG-VSF), a novel pretraining task that models forecasting as a conditional generation task, where driver variables (e.g., weather) inform the prediction of response variables (e.g., satellite imagery). We demonstrate that pretraining in such a fashion leads to strong embeddings which give enhanced performance when finetuned on downstream tasks where capturing this causality matters such as pixel wise crop type mapping, soil moisture estimation and forecasting, missing image prediction, and future image forecasting when compared to finetuning embeddings from other standard pretraining approaches.

翻译：自监督学习已成为利用大规模数据预训练基础模型的强大范式。现有预训练方法主要依赖于掩码重建或下一标记预测策略，在包括地球科学应用在内的各种下游任务中展现出优异性能。然而，这些方法未能充分捕捉不同地理空间与环境变量间因果相互作用的知识。为突破此局限，我们提出知识引导的变步长预测（KG-VSF）——一种将预测建模为条件生成任务的新型预训练方法，其中驱动变量（如气象数据）为响应变量（如卫星影像）的预测提供信息。我们证明，相较于其他标准预训练方法生成的嵌入进行微调，以此方式预训练的模型能产生强表征嵌入，在需要捕捉此类因果关系的下游任务（如逐像素作物类型分类、土壤湿度估算与预测、缺失影像预测及未来影像预测）微调时，可获得显著提升的性能表现。