A regression model with more parameters than data points in the training data is overparametrized and has the capability to interpolate the training data. Based on the classical bias-variance tradeoff expressions, it is commonly assumed that models which interpolate noisy training data are poor to generalize. In some cases, this is not true. The best models obtained are overparametrized and the testing error exhibits the double descent behavior as the model order increases. In this contribution, we provide some analysis to explain the double descent phenomenon, first reported in the machine learning literature. We focus on interpolating models derived from the minimum norm solution to the classical least-squares problem and also briefly discuss model fitting using ridge regression. We derive a result based on the behavior of the smallest singular value of the regression matrix that explains the peak location and the double descent shape of the testing error as a function of model order.

翻译：训练数据中参数数量超过数据点的回归模型属于过参数化模型，具备插值训练数据的能力。基于经典的偏差-方差权衡表达式，通常认为插值含噪训练数据的模型泛化能力较差。但在某些情况下，这一假设并不成立。最优模型往往呈现过参数化特征，且随着模型阶数增加，测试误差展现出双重下降行为。本文对机器学习文献中首次报道的双重下降现象进行了理论分析。我们重点研究基于经典最小二乘问题最小范数解的插值模型，并简要讨论岭回归的模型拟合方法。基于回归矩阵最小奇异值的行为特征，我们推导出一个能够解释测试误差随模型阶数变化时峰值位置与双重下降形态的结论。