Benign overfitting, the phenomenon where interpolating models generalize well in the presence of noisy data, was first observed in neural network models trained with gradient descent. To better understand this empirical observation, we consider the generalization error of two-layer neural networks trained to interpolation by gradient descent on the logistic loss following random initialization. We assume the data comes from well-separated class-conditional log-concave distributions and allow for a constant fraction of the training labels to be corrupted by an adversary. We show that in this setting, neural networks exhibit benign overfitting: they can be driven to zero training error, perfectly fitting any noisy training labels, and simultaneously achieve minimax optimal test error. In contrast to previous work on benign overfitting that require linear or kernel-based predictors, our analysis holds in a setting where both the model and learning dynamics are fundamentally nonlinear.

翻译：良性过拟合——即插值模型在含噪数据中仍能实现良好泛化的现象——最早在通过梯度下降训练的神经网络模型中被观察到。为深入理解这一实验发现，我们研究了在逻辑损失函数上通过随机初始化后梯度下降训练至插值状态的两层神经网络的泛化误差。假设数据来自具有良好分离性的类条件对数凹分布，并允许训练标签中存在恒定比例的对抗性噪声。我们证明在此设定下，神经网络表现出良性过拟合：模型可达到零训练误差，完美拟合任意含噪训练标签，同时实现极小极大最优测试误差。与早期要求线性或核方法的良性过拟合研究不同，我们的分析在模型和学习动态均为本质非线性的条件下成立。