Neural operator learning models have emerged as very effective surrogates in data-driven methods for partial differential equations (PDEs) across different applications from computational science and engineering. Such operator learning models not only predict particular instances of a physical or biological system in real-time but also forecast classes of solutions corresponding to a distribution of initial and boundary conditions or forcing terms. % DeepONet is the first neural operator model and has been tested extensively for a broad class of solutions, including Riemann problems. Transformers have not been used in that capacity, and specifically, they have not been tested for solutions of PDEs with low regularity. % In this work, we first establish the theoretical groundwork that transformers possess the universal approximation property as operator learning models. We then apply transformers to forecast solutions of diverse dynamical systems with solutions of finite regularity for a plurality of initial conditions and forcing terms. In particular, we consider three examples: the Izhikevich neuron model, the tempered fractional-order Leaky Integrate-and-Fire (LIF) model, and the one-dimensional Euler equation Riemann problem. For the latter problem, we also compare with variants of DeepONet, and we find that transformers outperform DeepONet in accuracy but they are computationally more expensive.

翻译：神经算子学习模型已成为计算科学与工程领域中数据驱动偏微分方程方法中极为有效的替代模型。此类算子学习模型不仅能实时预测物理或生物系统的特定实例，还能针对初始条件、边界条件或外力项分布对应的解类进行预测。% DeepONet是首个神经算子模型，已在包括黎曼问题在内的广泛解类中得到充分验证。Transformer尚未在此类能力中得到应用，尤其未在低正则性偏微分方程解的测试中得到验证。% 本研究首先建立理论基础，证明Transformer具备作为算子学习模型的通用逼近性质。随后将Transformer应用于多类初始条件与外力项下具有有限正则性解的动力系统预测。具体而言，我们研究了三个示例：Izhikevich神经元模型、分数阶泄漏积分点火模型的修正形式，以及一维欧拉方程黎曼问题。针对最后一项问题，我们与DeepONet的变体模型进行了对比，发现Transformer在精度上优于DeepONet，但计算成本更高。