We provide sharp path-dependent generalization and excess risk guarantees for the full-batch Gradient Descent (GD) algorithm on smooth losses (possibly non-Lipschitz, possibly nonconvex). At the heart of our analysis is an upper bound on the generalization error, which implies that average output stability and a bounded expected optimization error at termination lead to generalization. This result shows that a small generalization error occurs along the optimization path, and allows us to bypass Lipschitz or sub-Gaussian assumptions on the loss prevalent in previous works. For nonconvex, convex, and strongly convex losses, we show the explicit dependence of the generalization error in terms of the accumulated path-dependent optimization error, terminal optimization error, number of samples, and number of iterations. For nonconvex smooth losses, we prove that full-batch GD efficiently generalizes close to any stationary point at termination, and recovers the generalization error guarantees of stochastic algorithms with fewer assumptions. For smooth convex losses, we show that the generalization error is tighter than existing bounds for SGD (up to one order of error magnitude). Consequently the excess risk matches that of SGD for quadratically less iterations. Lastly, for strongly convex smooth losses, we show that full-batch GD achieves essentially the same excess risk rate as compared with the state of the art on SGD, but with an exponentially smaller number of iterations (logarithmic in the dataset size).

翻译：我们为平滑损失（可能非利普希茨、可能非凸）上的全批量梯度下降算法提供了依赖于路径的锐利泛化与超额风险保证。我们分析的核心是泛化误差的一个上界，该上界表明平均输出稳定性与终止时有限的最优化期望误差共同导致泛化。这一结果揭示了沿优化路径存在小的泛化误差，并使我们能够绕过先前研究中普遍存在的关于损失的利普希茨或次高斯假设。针对非凸、凸和强凸损失，我们展示了泛化误差关于累积路径依赖优化误差、终止优化误差、样本数量与迭代次数的显式依赖关系。对于非凸平滑损失，我们证明全批量梯度下降在终止点附近的任意驻点处能有效泛化，并以更少的假设恢复了随机算法的泛化误差保证。对于光滑凸损失，我们证明其泛化误差比现有随机梯度下降的界更紧（误差量级至多提升一个数量级），进而使超额风险在迭代次数平方根减少的情况下与随机梯度下降持平。最后，对于强凸光滑损失，我们证明全批量梯度下降能达到与当前最优随机梯度下降方法几乎相同的超额风险率，但迭代次数呈指数级减少（数据集大小的对数阶）。