Pre-training has proven effective in addressing data scarcity and performance limitations in solving PDE problems with neural operators. However, challenges remain due to the heterogeneity of PDE datasets in equation types, which leads to high errors in mixed training. Additionally, dense pre-training models that scale parameters by increasing network width or depth incur significant inference costs. To tackle these challenges, we propose a novel Mixture-of-Experts Pre-training Operator Transformer (MoE-POT), a sparse-activated architecture that scales parameters efficiently while controlling inference costs. Specifically, our model adopts a layer-wise router-gating network to dynamically select 4 routed experts from 16 expert networks during inference, enabling the model to focus on equation-specific features. Meanwhile, we also integrate 2 shared experts, aiming to capture common properties of PDE and reduce redundancy among routed experts. The final output is computed as the weighted average of the results from all activated experts. We pre-train models with parameters from 30M to 0.5B on 6 public PDE datasets. Our model with 90M activated parameters achieves up to a 40% reduction in zero-shot error compared with existing models with 120M activated parameters. Additionally, we conduct interpretability analysis, showing that dataset types can be inferred from router-gating network decisions, which validates the rationality and effectiveness of the MoE architecture.

翻译：预训练已被证明能有效缓解神经算子在求解偏微分方程问题中面临的数据稀缺与性能限制。然而，由于偏微分方程数据集在方程类型上存在异质性，导致混合训练时误差较高，这仍是当前面临的挑战。此外，通过增加网络宽度或深度来扩展参数的密集预训练模型会产生显著的推理成本。为解决这些问题，我们提出了一种新颖的专家混合预训练算子Transformer（MoE-POT），这是一种稀疏激活架构，能在控制推理成本的同时高效扩展参数。具体而言，我们的模型采用分层路由门控网络，在推理过程中从16个专家网络中动态选择4个路由专家，使模型能够聚焦于方程特定的特征。同时，我们还整合了2个共享专家，旨在捕获偏微分方程的共性特征并减少路由专家之间的冗余。最终输出通过所有激活专家结果的加权平均计算得出。我们在6个公开偏微分方程数据集上对参数规模从3000万到5亿的模型进行了预训练。我们激活参数为9000万的模型，相比现有激活参数为1.2亿的模型，在零样本误差上最高可降低40%。此外，我们进行了可解释性分析，结果表明可以从路由门控网络的决策中推断数据集类型，这验证了MoE架构的合理性与有效性。