Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.

翻译：物理信息神经网络（PINNs）近年来已成为深度神经网络在非线性偏微分方程（PDE）数值求解中的一个重要应用。然而，人们已认识到需要采用自适应方法，迫使神经网络准确拟合"刚性"PDE解中的顽固区域。本文提出了一种全新的自适应训练PINNs的方法，其中自适应权重完全可训练，并单独应用于每个训练点，从而使神经网络能够自主识别解中难以拟合的区域，并迫使网络专注于这些区域。该自适应权重定义了一个软乘性注意力掩码，其机制类似于计算机视觉中使用的注意力机制。SA-PINNs的基本原理是使权重随相应损失值的增大而增加，这通过同时训练网络最小化损失和最大化权重来实现。此外，我们展示了如何利用高斯过程回归构建连续的自适应权重映射，这使得在传统梯度下降法无法获得精确解的问题中能够使用随机梯度下降法。最后，我们推导了SA-PINNs的神经正切核矩阵，并利用该矩阵在无限宽PINNs的极限情况下，启发性地理解了自适应权重对训练动态的影响，这表明SA-PINNs通过平滑均衡不同损失项对应的NTK矩阵特征值来发挥作用。在多个线性和非线性基准问题的数值实验中，SA-PINNs在L2误差方面优于其他最先进的PINN算法，且使用的训练周期数更少。