We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility.

翻译：本文提出多物理场预训练（MPP），一种面向时空系统物理代理建模的自回归任务无关预训练方法，基于Transformer架构。在MPP中，我们不再针对特定物理系统训练单一模型，而是训练一个骨干模型同时预测多个异构物理系统的动力学行为，从而学习跨系统通用的特征表示并促进知识迁移。为在此设定下实现高效学习，我们提出共享嵌入与归一化策略，将多系统的物理场投影至共享嵌入空间。我们在涵盖广泛流体力学场景的基准测试中，验证了该方法在预训练及下游任务上的有效性。实验表明：单一经MPP预训练的Transformer无需微调即可在所有预训练子任务上达到或超越任务专用基线模型的性能。对于下游任务，我们证明相较于从头训练或微调预训练视频基础模型，对MPP训练后的模型进行微调，能够在包含未见物理组件或更高维度的系统上，实现多时间步预测精度的显著提升。为促进可复现性，我们开源了代码及多尺度训练所得的模型权重。