Simple stochastic momentum methods are widely used in machine learning optimization, but their good practical performance is at odds with an absence of theoretical guarantees of acceleration in the literature. In this work, we aim to close the gap between theory and practice by showing that stochastic heavy ball momentum retains the fast linear rate of (deterministic) heavy ball momentum on quadratic optimization problems, at least when minibatching with a sufficiently large batch size. The algorithm we study can be interpreted as an accelerated randomized Kaczmarz algorithm with minibatching and heavy ball momentum. The analysis relies on carefully decomposing the momentum transition matrix, and using new spectral norm concentration bounds for products of independent random matrices. We provide numerical illustrations demonstrating that our bounds are reasonably sharp.

翻译：简单随机动量方法广泛应用于机器学习优化中，但其良好的实际性能与文献中缺乏加速的理论保证之间存在矛盾。本研究旨在弥合理论与实践的差距，通过证明随机重球动量法在二次优化问题上保持（确定性）重球动量法的快速线性收敛速率，至少在使用足够大批量的小批量采样时成立。我们研究的算法可解释为一种融合了小批量采样和重球动量法的加速随机Kaczmarz算法。该分析依赖于对动量转移矩阵的精细分解，以及针对独立随机矩阵乘积的新谱范数集中界。我们提供了数值示例，证明所得边界具有合理的紧致性。