Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the type of overfitting in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.

翻译：过参数化神经网络（NNs）即使在完美拟合含噪数据训练时，也能展现出良好的泛化能力。这一现象催生了大量关于"良性过拟合"的研究工作，其中插值预测器能够达到近最优性能。近期，有研究推测并通过实验观察到，神经网络的行为通常更适合用"适度过拟合"来描述——其性能虽非最优但也非平凡，且随噪声水平增加而退化。然而，目前对于非线性神经网络这一论断的理论证明仍然缺失。在本工作中，我们提出了若干旨在弥合这两种互补视角的研究成果。我们研究了包含两层ReLU神经网络的简单分类场景，并证明在各种假设条件下，过拟合类型会从极端一维数据情况下的适度过拟合，转变为高维数据下的良性过拟合。因此，我们揭示了输入维度在此设定中对过拟合类型的关键作用，并通过中间维度的实证研究进一步验证了这一发现。总体而言，我们的研究结果阐明了维度、样本量、网络架构及训练算法与最终过拟合类型之间错综复杂的关联关系。