We propose energy natural gradient descent, a natural gradient method with respect to a Hessian-induced Riemannian metric as an optimization algorithm for physics-informed neural networks (PINNs) and the deep Ritz method. As a main motivation we show that the update direction in function space resulting from the energy natural gradient corresponds to the Newton direction modulo an orthogonal projection onto the model's tangent space. We demonstrate experimentally that energy natural gradient descent yields highly accurate solutions with errors several orders of magnitude smaller than what is obtained when training PINNs with standard optimizers like gradient descent or Adam, even when those are allowed significantly more computation time.

翻译：我们提出能量自然梯度下降法，这是一种基于海森矩阵诱导的黎曼度量的自然梯度方法，可作为物理信息神经网络（PINNs）和深度里茨方法的优化算法。主要动机在于证明：能量自然梯度在函数空间中产生的更新方向，等价于在模型切空间上进行正交投影后的牛顿方向。实验表明，即使允许梯度下降或Adam等标准优化器使用显著更多的计算时间，能量自然梯度下降法仍能获得高精度解，其误差比使用标准优化器训练PINNs时小数个数量级。