Accelerated stochastic gradient descent (ASGD) is a workhorse in deep learning and often achieves better generalization performance than SGD. However, existing optimization theory can only explain the faster convergence of ASGD, but cannot explain its better generalization. In this paper, we study the generalization of ASGD for overparameterized linear regression, which is possibly the simplest setting of learning with overparameterization. We establish an instance-dependent excess risk bound for ASGD within each eigen-subspace of the data covariance matrix. Our analysis shows that (i) ASGD outperforms SGD in the subspace of small eigenvalues, exhibiting a faster rate of exponential decay for bias error, while in the subspace of large eigenvalues, its bias error decays slower than SGD; and (ii) the variance error of ASGD is always larger than that of SGD. Our result suggests that ASGD can outperform SGD when the difference between the initialization and the true weight vector is mostly confined to the subspace of small eigenvalues. Additionally, when our analysis is specialized to linear regression in the strongly convex setting, it yields a tighter bound for bias error than the best-known result.

翻译：加速随机梯度下降法（ASGD）是深度学习中的核心算法，其泛化性能通常优于标准随机梯度下降法（SGD）。然而，现有优化理论仅能解释ASGD的快速收敛特性，却无法解释其更优的泛化能力。本文在过参数化线性回归这一最简单的过参数化学习框架下，系统研究了ASGD的泛化性能。我们在数据协方差矩阵的每个特征子空间上建立了ASGD的实例依赖超额风险界。分析表明：(i) 在特征值较小的子空间中，ASGD表现优于SGD，其偏差误差呈更快指数衰减速率，而在特征值较大的子空间中，其偏差误差衰减慢于SGD；(ii) ASGD的方差误差始终大于SGD。研究结果表明：当初始参数与真实权值向量的差异主要集中于小特征值子空间时，ASGD可超越SGD。此外，将本分析特化至强凸线性回归场景时，所得偏差误差上界比当前已知最优结果更为紧致。