It is known that the standard stochastic gradient descent (SGD) optimization method, as well as accelerated and adaptive SGD optimization methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (as, for example, in the situation of constant learning rates). Numerical simulations often use human-tuned deterministic learning rate schedules or small constant learning rates. The default learning rate schedules for SGD optimization methods in machine learning implementation frameworks such as TensorFlow and Pytorch are constant learning rates. In this work we propose and study a learning-rate-adaptive approach for SGD optimization methods in which the learning rate is adjusted based on empirical estimates for the values of the objective function of the considered optimization problem (the function that one intends to minimize). In particular, we propose a learning-rate-adaptive variant of the Adam optimizer and implement it in case of several neural network learning problems, particularly, in the context of deep learning approximation methods for partial differential equations such as deep Kolmogorov methods, physics-informed neural networks, and deep Ritz methods. In each of the presented learning problems the proposed learning-rate-adaptive variant of the Adam optimizer faster reduces the value of the objective function than the Adam optimizer with the default learning rate. For a simple class of quadratic minimization problems we also rigorously prove that a learning-rate-adaptive variant of the SGD optimization method converges to the minimizer of the considered minimization problem. Our convergence proof is based on an analysis of the laws of invariant measures of the SGD method as well as on a more general convergence analysis for SGD with random but predictable learning rates which we develop in this work.

翻译：众所周知，标准随机梯度下降优化方法以及加速和自适应SGD优化方法（如Adam优化器）在学习率不收敛于零时（例如采用恒定学习率的情况）会无法收敛。数值模拟通常采用人工调谐的确定性学习率调度或较小的恒定学习率。在TensorFlow和PyTorch等机器学习实现框架中，SGD优化方法的默认学习率调度均为恒定学习率。本研究提出并探讨了一种SGD优化方法的学习率自适应策略，该策略基于对优化问题目标函数值（待最小化函数）的经验估计来调整学习率。特别地，我们提出了Adam优化器的学习率自适应变体，并在若干神经网络学习问题中进行了实现，尤其是在偏微分方程的深度学习近似方法背景下，包括深度Kolmogorov方法、物理信息神经网络以及深度Ritz方法。在所有展示的学习问题中，所提出的学习率自适应Adam变体比采用默认学习率的Adam优化器能更快降低目标函数值。针对一类简单的二次最小化问题，我们严格证明了SGD优化方法的学习率自适应变体能收敛至所考虑最小化问题的最优解。我们的收敛性证明基于对SGD方法不变测度规律的分析，以及本工作中提出的针对具有随机但可预测学习率的SGD的更广义收敛性分析。