Studies on benign overfitting provide insights for the success of overparameterized deep learning models. In this work, we examine whether overfitting is truly benign in real-world classification tasks. We start with the observation that a ResNet model overfits benignly on Cifar10 but not benignly on ImageNet. To understand why benign overfitting fails in the ImageNet experiment, we theoretically analyze benign overfitting under a more restrictive setup where the number of parameters is not significantly larger than the number of data points. Under this mild overparameterization setup, our analysis identifies a phase change: unlike in the previous heavy overparameterization settings, benign overfitting can now fail in the presence of label noise. Our analysis explains our empirical observations, and is validated by a set of control experiments with ResNets. Our work highlights the importance of understanding implicit bias in underfitting regimes as a future direction.

翻译：关于良性过拟合的研究为过参数化深度学习模型成功提供了见解。本文探讨实际分类任务中过拟合是否真正良性。我们首先观察到ResNet模型在Cifar10数据集上呈现良性过拟合，但在ImageNet上并非如此。为理解ImageNet实验中良性过拟合失效的原因，我们在参数数量未显著超过数据点数量的更严格设置下进行了理论分析。在此轻度过参数化设定下，我们的分析识别出一个相变现象：与先前重度过参数化设置不同，存在标签噪声时良性过拟合可能失效。该分析解释了我们的经验观察，并通过一组ResNet控制实验得到验证。本文强调理解欠拟合机制中隐式偏差作为未来研究方向的重要性。