The success of over-parameterized neural networks trained to near-zero training error has caused great interest in the phenomenon of benign overfitting, where estimators are statistically consistent even though they interpolate noisy training data. While benign overfitting in fixed dimension has been established for some learning methods, current literature suggests that for regression with typical kernel methods and wide neural networks, benign overfitting requires a high-dimensional setting where the dimension grows with the sample size. In this paper, we show that the smoothness of the estimators, and not the dimension, is the key: benign overfitting is possible if and only if the estimator's derivatives are large enough. We generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension. Conversely, we show that benign overfitting is possible for regression with a sequence of spiky-smooth kernels with large derivatives. Using neural tangent kernels, we translate our results to wide neural networks. We prove that while infinite-width networks do not overfit benignly with the ReLU activation, this can be fixed by adding small high-frequency fluctuations to the activation function. Our experiments verify that such neural networks, while overfitting, can indeed generalize well even on low-dimensional data sets.

翻译：超参数化神经网络在训练至近乎零误差时取得的成功引起了人们对“良性过拟合”现象的极大兴趣——尽管估计器插值了含噪声的训练数据，它们仍具有统计一致性。尽管对于某些学习方法，已在固定维度下确立了良性过拟合的存在，但当前文献表明，对于使用典型核方法和宽神经网络的回归问题，良性过拟合需要维度随样本量增长的高维设置。本文表明，关键是估计器的光滑性而非维度：当且仅当估计器的导数足够大时，良性过拟合才可能实现。我们将现有的不一致性结果推广至非插值模型及更多核，证明在固定维度下，具有中等导数的良性过拟合是不可能的。反之，我们证明对于具有大导数的一系列尖峰光滑核的回归问题，良性过拟合是可能的。利用神经正切核，我们将结果推广至宽神经网络。我们证明，虽然采用ReLU激活函数的无穷宽网络无法实现良性过拟合，但通过向激活函数添加微小的高频波动可解决此问题。实验验证表明，此类神经网络在过拟合的同时，即使在低维数据集上也能具有良好的泛化能力。