In multi-objective optimization, multiple loss terms are weighted and added together to form a single objective. These weights are chosen to properly balance the competing losses according to some meta-goal. For example, in physics-informed neural networks (PINNs), these weights are often adaptively chosen to improve the network's generalization error. A popular choice of adaptive weights is based on the neural tangent kernel (NTK) of the PINN, which describes the evolution of the network in predictor space during training. The convergence of such an adaptive weighting algorithm is not clear a priori. Moreover, these NTK-based weights would be updated frequently during training, further increasing the computational burden of the learning process. In this paper, we prove that under appropriate conditions, gradient descent enhanced with adaptive NTK-based weights is convergent in a suitable sense. We then address the problem of computational efficiency by developing a randomized algorithm inspired by a predictor-corrector approach and matrix sketching, which produces unbiased estimates of the NTK up to an arbitrarily small discretization error. Finally, we provide numerical experiments to support our theoretical findings and to show the efficacy of our randomized algorithm. Code Availability: https://github.com/maxhirsch/Efficient-NTK

翻译：在多目标优化中，多个损失项通过加权求和构成单一目标函数。这些权重通常根据某种元目标被选取，以适当平衡相互竞争的损失项。例如，在物理信息神经网络（PINNs）中，这些权重常被自适应地选择以改善网络的泛化误差。一种常用的自适应权重选择方法基于PINN的神经正切核（NTK），该核描述了训练过程中网络在预测空间中的演化。此类自适应加权算法的收敛性在理论上并非先验明确。此外，这些基于NTK的权重在训练过程中需频繁更新，进一步增加了学习过程的计算负担。本文证明了在适当条件下，采用基于NTK的自适应权重增强的梯度下降法在特定意义下是收敛的。随后，我们通过受预测-校正方法和矩阵草图法启发的随机化算法解决了计算效率问题，该算法能以任意小的离散化误差无偏估计NTK。最后，我们提供了数值实验以支持理论结果，并展示了所提随机化算法的有效性。代码可用性：https://github.com/maxhirsch/Efficient-NTK