There is a notable dearth of results characterizing the preconditioning effect of Adam and showing how it may alleviate the curse of ill-conditioning -- an issue plaguing gradient descent (GD). In this work, we perform a detailed analysis of Adam's preconditioning effect for quadratic functions and quantify to what extent Adam can mitigate the dependence on the condition number of the Hessian. Our key finding is that Adam can suffer less from the condition number but at the expense of suffering a dimension-dependent quantity. Specifically, for a $d$-dimensional quadratic with a diagonal Hessian having condition number $\kappa$, we show that the effective condition number-like quantity controlling the iteration complexity of Adam without momentum is $\mathcal{O}(\min(d, \kappa))$. For a diagonally dominant Hessian, we obtain a bound of $\mathcal{O}(\min(d \sqrt{d \kappa}, \kappa))$ for the corresponding quantity. Thus, when $d < \mathcal{O}(\kappa^p)$ where $p = 1$ for a diagonal Hessian and $p = 1/3$ for a diagonally dominant Hessian, Adam can outperform GD (which has an $\mathcal{O}(\kappa)$ dependence). On the negative side, our results suggest that Adam can be worse than GD for a sufficiently non-diagonal Hessian even if $d \ll \mathcal{O}(\kappa^{1/3})$; we corroborate this with empirical evidence. Finally, we extend our analysis to functions satisfying per-coordinate Lipschitz smoothness and a modified version of the Polyak-\L ojasiewicz condition.

翻译：关于亚当优化器预处理效应的表征及其如何缓解梯度下降法中病态条件数问题的研究结果仍显不足。本文针对二次型函数深入分析了亚当优化器的预处理效应，量化了其在何种程度上能减轻对海森矩阵条件数的依赖。关键发现是：亚当优化器受条件数影响较小，但代价是需要承受维度相关量。具体而言，对于具有对角海森矩阵且条件数为$\kappa$的$d$维二次型函数，我们证明控制无动量亚当优化器迭代复杂度的有效条件数型量级为$\mathcal{O}(\min(d, \kappa))$。对于对角占优海森矩阵，对应量级边界为$\mathcal{O}(\min(d \sqrt{d \kappa}, \kappa))$。因此当$d < \mathcal{O}(\kappa^p)$（对角海森矩阵$p=1$，对角占优海森矩阵$p=1/3$）时，亚当优化器可优于梯度下降法（其复杂度依赖关系为$\mathcal{O}(\kappa)$）。从负面角度看，我们的结果表明：即使$d \ll \mathcal{O}(\kappa^{1/3})$，对于非对角性较强的海森矩阵，亚当优化器可能逊于梯度下降法——我们通过实证验证了这一现象。最后，我们将分析扩展至满足逐坐标利普希茨光滑性及修正版Polyak-Łojasiewicz条件的函数。