Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems, but they are still trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit stochastic gradient descent (ISGD) method to train PINNs for improving the stability of training process. We heuristically analyze how ISGD overcome stiffness in the gradient flow dynamics of PINNs, especially for problems with multi-scale solutions. We theoretically prove that for two-layer fully connected neural networks with large hidden nodes, randomly initialized ISGD converges to a globally optimal solution for the quadratic loss function. Empirical results demonstrate that ISGD works well in practice and compares favorably to other gradient-based optimization methods such as SGD and Adam, while can also effectively address the numerical stiffness in training dynamics via gradient descent.

翻译：物理信息神经网络（PINNs）已在求解正向和逆向微分方程问题中得到有效验证，但当待逼近的目标函数呈现高频或多尺度特征时，仍会陷入训练失败。本文提出采用隐式随机梯度下降法（ISGD）训练PINNs以提升训练过程的稳定性。我们通过启发式分析揭示了ISGD如何克服PINNs梯度流动力学中的刚性问题，尤其是针对多尺度解问题。理论证明，对于具有大量隐藏节点的两层全连接神经网络，随机初始化的ISGD在二次损失函数下能收敛至全局最优解。实验结果表明，ISGD在实践中表现优异，且优于SGD和Adam等基于梯度的优化方法，同时能有效解决梯度下降训练动力学中的数值刚性问题。