For autoregressive modeling of chaotic dynamical systems over long time horizons, the stability of both training and inference is a major challenge in building scientific foundation models. We present a hybrid technique in which an autoregressive transformer is embedded within a novel shooting-based mixed finite element scheme, exposing topological structure that enables provable stability. For forward problems, we prove preservation of discrete energies, while for training we prove uniform bounds on gradients, provably avoiding the exploding gradient problem. Combined with a vision transformer, this yields latent tokens admitting structure-preserving dynamics. We outperform modern foundation models with a $65\times$ reduction in model parameters and long-horizon forecasting of chaotic systems. A "mini-foundation" model of a fusion component shows that 12 simulations suffice to train a real-time surrogate, achieving a $9{,}000\times$ speedup over particle-in-cell simulation.

翻译：针对混沌动力系统长时域自回归建模，训练与推理的稳定性是构建科学基础模型的核心挑战。我们提出一种混合技术，将自回归Transformer嵌入新颖的基于打靶法的混合有限元格式中，通过暴露拓扑结构实现可证明的稳定性。对于前向问题，我们证明了离散能量的守恒性；对于训练过程，我们证明了梯度的均匀有界性，从而可证明地避免了梯度爆炸问题。结合视觉Transformer后，生成的潜层令牌可保持结构守恒动力学特性。我们以模型参数减少65倍的优势超越现代基础模型，并实现混沌系统的长时域预测。基于聚变组件构建的"微型基础"模型表明，仅需12次仿真即可训练出实时替代模型，相比粒子网格仿真实现9000倍加速。