We develop a high-dimensional scaling limit for Stochastic Gradient Descent with Polyak Momentum (SGD-M) and adaptive step-sizes. This provides a framework to rigourously compare online SGD with some of its popular variants. We show that the scaling limits of SGD-M coincide with those of online SGD after an appropriate time rescaling and a specific choice of step-size. However, if the step-size is kept the same between the two algorithms, SGD-M will amplify high-dimensional effects, potentially degrading performance relative to online SGD. We demonstrate our framework on two popular learning problems: Spiked Tensor PCA and Single Index Models. In both cases, we also examine online SGD with an adaptive step-size based on normalized gradients. In the high-dimensional regime, this algorithm yields multiple benefits: its dynamics admit fixed points closer to the population minimum and widens the range of admissible step-sizes for which the iterates converge to such solutions. These examples provide a rigorous account, aligning with empirical motivation, of how early preconditioners can stabilize and improve dynamics in settings where online SGD fails.

翻译：我们为带有Polyak动量的随机梯度下降（SGD-M）及自适应步长建立了高维标度极限。该框架为严格比较在线SGD与其若干常用变体提供了理论基础。我们证明，在适当的时间重标度及特定步长选择下，SGD-M的标度极限与在线SGD的标度极限一致。然而，若两种算法保持相同步长，SGD-M将放大高维效应，可能导致其性能相对于在线SGD下降。我们在两个经典学习问题上验证了该框架：尖峰张量主成分分析与单指标模型。针对这两种情形，我们还研究了基于归一化梯度的自适应步长在线SGD。在高维体系中，该算法展现出多重优势：其动力学允许更接近总体最小值的固定点，并拓宽了迭代收敛至此类解的可容许步长范围。这些实例为早期预条件子如何在在线SGD失效的场景中稳定并改善动力学过程提供了严格的理论解释，与经验动机相吻合。