This paper provides the first tight convergence analyses for RMSProp and Adam in non-convex optimization under the most relaxed assumptions of coordinate-wise generalized smoothness and affine noise variance. We first analyze RMSProp, which is a special case of Adam with adaptive learning rates but without first-order momentum. Specifically, to solve the challenges due to dependence among adaptive update, unbounded gradient estimate and Lipschitz constant, we demonstrate that the first-order term in the descent lemma converges and its denominator is upper bounded by a function of gradient norm. Based on this result, we show that RMSProp with proper hyperparameters converges to an $\epsilon$-stationary point with an iteration complexity of $\mathcal O(\epsilon^{-4})$. We then generalize our analysis to Adam, where the additional challenge is due to a mismatch between the gradient and first-order momentum. We develop a new upper bound on the first-order term in the descent lemma, which is also a function of the gradient norm. We show that Adam with proper hyperparameters converges to an $\epsilon$-stationary point with an iteration complexity of $\mathcal O(\epsilon^{-4})$. Our complexity results for both RMSProp and Adam match with the complexity lower bound established in \cite{arjevani2023lower}.

翻译：本文首次在坐标广义光滑性和仿射噪声方差的最宽松假设下，针对非凸优化中的RMSProp和Adam算法提供了严格的收敛性分析。我们首先分析RMSProp——该算法是自适应学习率且不含一阶动量的Adam特例。具体而言，为应对自适应更新依赖性、无界梯度估计及Lipschitz常数等挑战，我们证明了下降引理中一阶项的收敛性，且其分母被梯度范数的函数所上界约束。基于此结果，我们证明了在适当超参数设置下，RMSProp能以$\mathcal O(\epsilon^{-4})$的迭代复杂度收敛至$\epsilon$-稳定点。随后我们将分析推广至Adam，此时需额外处理梯度与一阶动量之间的失配问题。我们为下降引理中的一阶项建立了新的上界，该上界同样依赖于梯度范数。研究表明，适当超参数配置下的Adam能以$\mathcal O(\epsilon^{-4})$的迭代复杂度收敛至$\epsilon$-稳定点。本文给出的RMSProp与Adam的复杂度结果均匹配文献\cite{arjevani2023lower}中建立的复杂度下界。