Existing theory suggests that for linear regression problems categorized by capacity and source conditions, gradient descent (GD) is always minimax optimal, while both ridge regression and online stochastic gradient descent (SGD) are polynomially suboptimal for certain categories of such problems. Moving beyond minimax theory, this work provides instance-wise comparisons of the finite-sample risks for these algorithms on any well-specified linear regression problem. Our analysis yields three key findings. First, GD dominates ridge regression: with comparable regularization, the excess risk of GD is always within a constant factor of that of ridge, but ridge can be polynomially worse even when tuned optimally. Second, GD is incomparable with SGD. While it is known that for certain problems GD can be polynomially better than SGD, the reverse is also true: we construct problems, inspired by benign overfitting theory, where optimally stopped GD is polynomially worse. Finally, GD dominates SGD for a significant subclass of problems -- those with fast and continuously decaying covariance spectra -- which includes all problems satisfying the standard capacity condition.

翻译：现有理论表明，对于按容量和源条件分类的线性回归问题，梯度下降始终是极小化极大最优的，而岭回归和在线随机梯度下降在某些问题类别中则是多项式次优的。本文超越极小化极大理论，针对任意良好设定的线性回归问题，提供了这些算法在有限样本风险上的实例级比较。分析得出三个关键发现。首先，梯度下降优于岭回归：在正则化程度相当的情况下，梯度下降的过剩风险始终在岭回归的常数因子范围内，但即使经过最优调参，岭回归仍可能呈多项式级更差。其次，梯度下降与随机梯度下降不可比较。虽然已知在某些问题上梯度下降可比随机梯度下降呈多项式级更优，但反之亦然：受良性过拟合理论启发，我们构建了问题，其中最优停止的梯度下降呈多项式级更差。最后，对于显著子类问题——即具有快速且连续衰减协方差谱的问题，包括所有满足标准容量条件的问题——梯度下降优于随机梯度下降。