Linear classifiers and leaky ReLU networks trained by gradient flow on the logistic loss have an implicit bias towards solutions which satisfy the Karush--Kuhn--Tucker (KKT) conditions for margin maximization. In this work we establish a number of settings where the satisfaction of these KKT conditions implies benign overfitting in linear classifiers and in two-layer leaky ReLU networks: the estimators interpolate noisy training data and simultaneously generalize well to test data. The settings include variants of the noisy class-conditional Gaussians considered in previous work as well as new distributional settings where benign overfitting has not been previously observed. The key ingredient to our proof is the observation that when the training data is nearly-orthogonal, both linear classifiers and leaky ReLU networks satisfying the KKT conditions for their respective margin maximization problems behave like a nearly uniform average of the training examples.

翻译：梯度流在逻辑损失下训练的线性分类器与Leaky ReLU网络隐式偏向于满足边界最大化问题的Karush-Kuhn-Tucker（KKT）条件的解。本文建立了一系列场景，在这些场景中，KKT条件的满足意味着线性分类器与两层Leaky ReLU网络存在良态过拟合：估计量插值含噪训练数据的同时对测试数据具有良好的泛化性能。这些场景包括先前工作中考虑的高斯含噪类条件变体，以及尚未观察到良态过拟合的新分布场景。我们证明的关键在于观察到：当训练数据近似正交时，满足各自边界最大化问题KKT条件的线性分类器与Leaky ReLU网络的行为类似于训练样本的近乎均匀平均。