Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p=n. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves indeed fold back into more traditional convex shapes - providing a resolution to tensions between double descent and statistical intuition.

翻译：传统的统计学理论建立了模型复杂度与预测误差之间清晰的关系，通常表现为一条U形曲线，反映欠拟合与过拟合之间的转变。然而，受过度参数化神经网络成功的启发，近期有影响力的工作认为该理论普遍不完整，并引入了一个新阶段——当参数数量p超过样本量n时，测试误差出现第二次下降，这一现象被称为“双重下降”。尽管大部分关注自然集中在深度学习领域，但研究表明双重下降在非神经网络模型中也普遍存在，已知案例包括线性回归、决策树和梯度提升。本文中，我们重新审视围绕这些经典统计机器学习方法的证据，并质疑其中观察到的双重下降是否真正突破了传统U形复杂度-泛化曲线的边界。我们证明，一旦仔细考量双重下降图中横轴的绘制内容，便会发现其中隐含着多条参数数量增长的复杂度轴。我们表明第二次下降恰好（且仅）在这些潜在轴之间的转换发生之时和之处出现，因此其位置并非固有地与插值阈值p=n相关。随后，我们通过采用经典非参数统计学视角获得进一步洞见。我们将所研究的方法解释为平滑器，并提出一个广义度量，用于衡量它们在未见样本上使用的有效参数数量。利用该度量，我们发现其看似双重下降的曲线确实折回为更传统的凸形曲线——从而化解了双重下降与统计学直觉之间的张力。