We analyze the statistical properties of generalized cross-validation (GCV) and leave-one-out cross-validation (LOOCV) applied to early-stopped gradient descent (GD) in high-dimensional least squares regression. We prove that GCV is generically inconsistent as an estimator of the prediction risk of early-stopped GD, even for a well-specified linear model with isotropic features. In contrast, we show that LOOCV converges uniformly along the GD trajectory to the prediction risk. Our theory requires only mild assumptions on the data distribution and does not require the underlying regression function to be linear. Furthermore, by leveraging the individual LOOCV errors, we construct consistent estimators for the entire prediction error distribution along the GD trajectory and consistent estimators for a wide class of error functionals. This in particular enables the construction of pathwise prediction intervals based on GD iterates that have asymptotically correct nominal coverage conditional on the training data.

翻译：我们分析了广义交叉验证（GCV）和留一交叉验证（LOOCV）在高维最小二乘回归中应用于早停梯度下降（GD）的统计性质。我们证明，在具有各向同性特征的良设定线性模型下，GCV作为早停GD预测风险的估计量通常是不一致的。相反，我们证明LOOCV沿GD轨迹一致收敛于预测风险。我们的理论仅需对数据分布施加温和假设，且无需假设底层回归函数为线性。此外，通过利用个体LOOCV误差，我们构造了沿GD轨迹的整个预测误差分布的一致估计量，以及广泛误差泛函的一致估计量。这尤其使得基于GD迭代的路径预测区间得以构建，其在训练数据条件下具有渐近正确的名义覆盖率。