The literature on "benign overfitting" in overparameterized models has been mostly restricted to regression or binary classification; however, modern machine learning operates in the multiclass setting. Motivated by this discrepancy, we study benign overfitting in multiclass linear classification. Specifically, we consider the following training algorithms on separable data: (i) empirical risk minimization (ERM) with cross-entropy loss, which converges to the multiclass support vector machine (SVM) solution; (ii) ERM with least-squares loss, which converges to the min-norm interpolating (MNI) solution; and, (iii) the one-vs-all SVM classifier. First, we provide a simple sufficient deterministic condition under which all three algorithms lead to classifiers that interpolate the training data and have equal accuracy. When the data is generated from Gaussian mixtures or a multinomial logistic model, this condition holds under high enough effective overparameterization. We also show that this sufficient condition is satisfied under "neural collapse", a phenomenon that is observed in training deep neural networks. Second, we derive novel bounds on the accuracy of the MNI classifier, thereby showing that all three training algorithms lead to benign overfitting under sufficient overparameterization. Ultimately, our analysis shows that good generalization is possible for SVM solutions beyond the realm in which typical margin-based bounds apply.

翻译：关于过参数化模型中“良性过拟合”的文献大多局限于回归或二分类问题；然而，现代机器学习主要应用于多类场景。受这一差异启发，我们研究了多类线性分类中的良性过拟合。具体而言，我们考虑了可分离数据上的以下训练算法：（i）采用交叉熵损失的经验风险最小化（ERM），该方法收敛到多类支持向量机（SVM）解；（ii）采用最小二乘损失的ERM，该方法收敛到最小范数插值（MNI）解；以及（iii）一类对多类SVM分类器。首先，我们给出了一个简单的充分确定性条件，在该条件下，所有三种算法产生的分类器都能插值训练数据并具有相等的精度。当数据由高斯混合模型或多项逻辑斯蒂模型生成时，该条件在足够高的有效过参数化下成立。我们还证明了该充分条件在“神经坍缩”现象（训练深度神经网络时观察到的现象）下也成立。其次，我们推导了MNI分类器精度的新型上界，从而表明在充分过参数化下，所有三种训练算法都能导致良性过拟合。最终，我们的分析表明，即使超出典型基于间隔的界适用的范围，SVM解也能实现良好的泛化。