Physics-Informed Neural Networks (PINNs) have revolutionized the computation of PDE solutions by integrating partial differential equations (PDEs) into the neural network's training process as soft constraints, becoming an important component of the scientific machine learning (SciML) ecosystem. More recently, physics-informed Kolmogorv-Arnold networks (PIKANs) have also shown to be effective and comparable in accuracy with PINNs. In their current implementation, both PINNs and PIKANs are mainly optimized using first-order methods like Adam, as well as quasi-Newton methods such as BFGS and its low-memory variant, L-BFGS. However, these optimizers often struggle with highly non-linear and non-convex loss landscapes, leading to challenges such as slow convergence, local minima entrapment, and (non)degenerate saddle points. In this study, we investigate the performance of Self-Scaled BFGS (SSBFGS), Self-Scaled Broyden (SSBroyden) methods and other advanced quasi-Newton schemes, including BFGS and L-BFGS with different line search strategies approaches. These methods dynamically rescale updates based on historical gradient information, thus enhancing training efficiency and accuracy. We systematically compare these optimizers -- using both PINNs and PIKANs -- on key challenging linear, stiff, multi-scale and non-linear PDEs, including the Burgers, Allen-Cahn, Kuramoto-Sivashinsky, and Ginzburg-Landau equations. Our findings provide state-of-the-art results with orders-of-magnitude accuracy improvements without the use of adaptive weights or any other enhancements typically employed in PINNs. More broadly, our results reveal insights into the effectiveness of second-order optimization strategies in significantly improving the convergence and accurate generalization of PINNs and PIKANs.

翻译：物理信息神经网络（PINNs）通过将偏微分方程（PDEs）作为软约束整合到神经网络的训练过程中，彻底改变了PDE求解的计算方式，已成为科学机器学习（SciML）生态系统的重要组成部分。最近，物理信息Kolmogorov-Arnold网络（PIKANs）也被证明是有效的，其精度与PINNs相当。在当前的实现中，PINNs和PIKANs主要使用Adam等一阶方法以及BFGS及其低内存变体L-BFGS等拟牛顿方法进行优化。然而，这些优化器在处理高度非线性和非凸的损失函数曲面时常常遇到困难，导致收敛缓慢、陷入局部极小值以及（非）退化鞍点等挑战。在本研究中，我们考察了自缩放BFGS（SSBFGS）、自缩放Broyden（SSBroyden）方法以及其他先进的拟牛顿方案（包括采用不同线搜索策略的BFGS和L-BFGS）的性能。这些方法基于历史梯度信息动态地重新缩放更新，从而提高了训练效率和精度。我们系统地比较了这些优化器——在关键的具有挑战性的线性、刚性、多尺度和非线性PDE上，使用PINNs和PIKANs进行了测试，包括Burgers、Allen-Cahn、Kuramoto-Sivashinsky和Ginzburg-Landau方程。我们的研究结果提供了最先进的成果，在不使用自适应权重或PINNs中通常采用的其他增强技术的情况下，实现了数量级的精度提升。更广泛地说，我们的结果揭示了二阶优化策略在显著改善PINNs和PIKANs的收敛性和准确泛化能力方面的有效性。