Pre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV). Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre-trained model size is scaled, (ii) the downstream training dataset size is scaled, (iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that-when fine-tuned appropriately-transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behavior across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential of the "pre-train and fine-tune" paradigm for SciML problems, demonstrating a path towards building SciML foundation models. We open-source our code for reproducibility.

翻译：预训练机器学习模型已在自然语言处理和计算机视觉等广泛领域展现出卓越性能。本研究旨在探索预训练机制在科学机器学习领域的应用潜力，重点关注迁移学习场景。我们系统研究了此类模型的迁移行为，具体考察维度包括：(i) 预训练模型规模的扩展效应；(ii) 下游训练数据集规模的缩放特性；(iii) 物理参数系统性偏离分布时的表现；以及(iv) 单一模型在混合多物理问题预训练后适配不同下游任务的泛化能力。研究表明：经适当微调后，迁移学习能以数量级更少的下游样本量（涵盖分布内乃至分布外任务），达到与从头训练相当的精度水平，且该行为在各类下游任务中保持高度一致性。此外，相较于在新下游任务上从头训练，通过微调预训练模型可获得随模型规模递增的渐进性能增益。上述结论在偏微分方程学习任务谱系中具有普适性。综合而言，本研究揭示了"预训练-微调"范式在科学机器学习问题中的巨大潜力，为构建科学机器学习基座模型指明了可行路径。为保障可复现性，我们已将相关代码开源。