Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions for partial differential equations (PDEs). While conventional PINNs and most related studies adopt fully-connected multilayer perceptrons (MLP) as the backbone structure, they have neglected the temporal relations in PDEs and failed to approximate the true solution. In this paper, we propose a novel Transformer-based framework, namely PINNsFormer, that accurately approximates PDEs' solutions by capturing the temporal dependencies with multi-head attention mechanisms in Transformer-based models. Instead of approximating point predictions, PINNsFormer adapts input vectors to pseudo sequences and point-wise PINNs loss to a sequential PINNs loss. In addition, PINNsFormer is equipped with a novel activation function, namely Wavelet, which anticipates the Fourier decomposition through deep neural networks. We empirically demonstrate PINNsFormer's ability to capture the PDE solutions for various scenarios, in which conventional PINNs have failed to learn. We also show that PINNsFormer achieves superior approximation accuracy on such problems than conventional PINNs with non-sensitive hyperparameters, in trade of marginal computational and memory costs, with extensive experiments.

翻译：物理信息神经网络（PINNs）已成为一种有前景的深度学习框架，用于逼近偏微分方程（PDEs）的数值解。尽管传统PINNs及大多数相关研究采用全连接多层感知机（MLP）作为骨干结构，但它们忽略了PDEs中的时间关系，未能准确逼近真实解。本文提出一种新颖的基于Transformer的框架，即PINNsFormer，通过利用Transformer模型中的多头注意力机制捕获时间依赖性，从而精确逼近PDEs的解。PINNsFormer不直接逼近逐点预测，而是将输入向量适配为伪序列，并将逐点PINNs损失转化为序列化PINNs损失。此外，PINNsFormer配备了一种新颖的激活函数——Wavelet，该函数通过深度神经网络预期实现傅里叶分解。我们通过实验证明，PINNsFormer能够捕获传统PINNs无法学习的各种场景下的PDE解。同时，通过大量实验表明，PINNsFormer在非敏感超参数设置下，以边际计算和内存成本为代价，在此类问题上实现了优于传统PINNs的逼近精度。