Stochastic gradient descent (SGD) is a workhorse algorithm for solving large-scale optimization problems in data science and machine learning. Understanding the convergence of SGD is hence of fundamental importance. In this work we examine the SGD convergence (with various step sizes) when applied to unconstrained convex quadratic programming (essentially least-squares (LS) problems), and in particular analyze the error components respect to the eigenvectors of the Hessian. The main message is that the convergence depends largely on the corresponding eigenvalues (singular values of the coefficient matrix in the LS context), namely the components for the large singular values converge faster in the initial phase. We then show there is a phase transition in the convergence where the convergence speed of the components, especially those corresponding to the larger singular values, will decrease. Finally, we show that the convergence of the overall error (in the solution) tends to decay as more iterations are run, that is, the initial convergence is faster than the asymptote.

翻译：随机梯度下降（SGD）是数据科学与机器学习领域解决大规模优化问题的核心算法，因此理解其收敛性具有根本重要性。本研究考察了SGD（采用多种步长设置）应用于无约束凸二次规划（本质上是最小二乘问题）时的收敛行为，特别针对Hessian矩阵特征向量方向的误差分量进行了分析。核心结论表明：收敛过程主要取决于对应的特征值（在最小二乘问题中即系数矩阵的奇异值），具体表现为较大奇异值对应的分量在迭代初期收敛更快。我们进一步揭示了收敛过程中存在相变现象，此时各分量（尤其是对应较大奇异值的分量）的收敛速度将显著下降。最后，我们证明随着迭代次数增加，整体解误差的收敛速率趋于衰减，即初始阶段的收敛速度快于渐近阶段。