In this paper, we provide sufficient conditions of benign overfitting of fixed width leaky ReLU two-layer neural network classifiers trained on mixture data via gradient descent. Our results are derived by establishing directional convergence of the network parameters and classification error bound of the convergent direction. Our classification error bound also lead to the discovery of a newly identified phase transition. Previously, directional convergence in (leaky) ReLU neural networks was established only for gradient flow. Due to the lack of directional convergence, previous results on benign overfitting were limited to those trained on nearly orthogonal data. All of our results hold on mixture data, which is a broader data setting than the nearly orthogonal data setting in prior work. We demonstrate our findings by showing that benign overfitting occurs with high probability in a much wider range of scenarios than previously known. Our results also allow us to characterize cases when benign overfitting provably fails even if directional convergence occurs. Our work thus provides a more complete picture of benign overfitting in leaky ReLU two-layer neural networks.

翻译：本文针对通过梯度下降在混合数据上训练的固定宽度 leaky ReLU 双层神经网络分类器，给出了其发生良性过拟合的充分条件。我们的结果通过建立网络参数的方向收敛性以及收敛方向的分类误差界而得到。我们的分类误差界还揭示了一种新发现的相变现象。此前，(leaky) ReLU 神经网络中的方向收敛性仅在梯度流条件下得到证明。由于缺乏方向收敛性，先前关于良性过拟合的结果仅限于在近似正交数据上训练的网络。我们的所有结果均在混合数据上成立，这是一个比先前工作中使用的近似正交数据设定更广泛的数据设定。我们通过证明良性过拟合以高概率发生在比以往所知更广泛的场景中，来展示我们的发现。我们的结果也使我们能够刻画即使方向收敛发生、但良性过拟合可证明失败的案例。因此，我们的工作为 leaky ReLU 双层神经网络中的良性过拟合提供了更完整的图景。