Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This paper focuses on exploring hyperparameter settings for the convergence of vanilla Adam and tackling the challenges of non-ergodic convergence related to practical application. The primary contributions are summarized as follows: firstly, we introduce precise definitions of ergodic and non-ergodic convergence, which cover nearly all forms of convergence for stochastic optimization algorithms. Meanwhile, we emphasize the superiority of non-ergodic convergence over ergodic convergence. Secondly, we establish a weaker sufficient condition for the ergodic convergence guarantee of Adam, allowing a more relaxed choice of hyperparameters. On this basis, we achieve the almost sure ergodic convergence rate of Adam, which is arbitrarily close to $o(1/\sqrt{K})$. More importantly, we prove, for the first time, that the last iterate of Adam converges to a stationary point for non-convex objectives. Finally, we obtain the non-ergodic convergence rate of $O(1/K)$ for function values under the Polyak-Lojasiewicz (PL) condition. These findings build a solid theoretical foundation for Adam to solve non-convex stochastic optimization problems.

翻译：Adam是机器学习中常用的随机优化算法。然而，其收敛性尚未被完全理解，尤其是在非凸设定下。本文聚焦于探索标准Adam在收敛过程中的超参数设置，并解决与实际应用相关的非遍历收敛难题。主要贡献总结如下：首先，我们给出了遍历收敛与非遍历收敛的精确定义，这涵盖了随机优化算法几乎所有形式的收敛性。同时，我们强调了非遍历收敛相对于遍历收敛的优越性。其次，我们建立了Adam遍历收敛保证的一个更弱充分条件，从而允许对超参数进行更宽松的选择。在此基础上，我们实现了Adam的几乎必然遍历收敛率，该速率可以任意接近$o(1/\sqrt{K})$。更为重要的是，我们首次证明了Adam的最后迭代点对于非凸目标能够收敛至平稳点。最后，在Polyak-Lojasiewicz (PL)条件下，我们获得了函数值的非遍历收敛率$O(1/K)$。这些发现为Adam求解非凸随机优化问题奠定了坚实的理论基础。