Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the type of overfitting in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.

翻译：过参数化的神经网络（NNs）即使在完美拟合含噪数据时，也展现出良好的泛化能力。这一现象催生了大量关于“良性过拟合”的研究——其中插值预测器能够实现接近最优的性能。近期，有学者推测并通过实验观察到，神经网络的行为通常更适合描述为“有节过拟合”，即性能虽非最优但亦非平凡，且会随噪声水平增加而退化。然而，目前尚缺乏对这一论断在非线性神经网络中理论上的合理解释。在本工作中，我们提供了一系列旨在弥合这两种互补视角的研究成果。我们研究了采用双层ReLU神经网络的简单分类设定，并证明在不同假设条件下，过拟合类型会从极端一维数据情形下的有节过拟合，过渡到高维情形下的良性过拟合。因此，我们揭示了输入维度在此设定中对过拟合类型的关键作用，并通过中间维度的实证研究验证了这一结论。总体而言，我们的研究阐明了维度、样本量、架构和训练算法与最终过拟合类型之间错综复杂的关联。