Successful deep learning models often involve training neural network architectures that contain more parameters than the number of training samples. Such overparametrized models have been extensively studied in recent years, and the virtues of overparametrization have been established from both the statistical perspective, via the double-descent phenomenon, and the computational perspective via the structural properties of the optimization landscape. Despite the remarkable success of deep learning architectures in the overparametrized regime, it is also well known that these models are highly vulnerable to small adversarial perturbations in their inputs. Even when adversarially trained, their performance on perturbed inputs (robust generalization) is considerably worse than their best attainable performance on benign inputs (standard generalization). It is thus imperative to understand how overparametrization fundamentally affects robustness. In this paper, we will provide a precise characterization of the role of overparametrization on robustness by focusing on random features regression models (two-layer neural networks with random first layer weights). We consider a regime where the sample size, the input dimension and the number of parameters grow in proportion to each other, and derive an asymptotically exact formula for the robust generalization error when the model is adversarially trained. Our developed theory reveals the nontrivial effect of overparametrization on robustness and indicates that for adversarially trained random features models, high overparametrization can hurt robust generalization.

翻译：成功的深度学习模型通常涉及训练参数数量超过训练样本数量的神经网络架构。近年来，这类过参数化模型得到了广泛研究，其优势已从统计视角（通过双下降现象）和计算视角（通过优化景观的结构特性）两方面得以确立。尽管深度学习架构在过参数化机制下取得了显著成功，但众所周知，这些模型极易受到输入中微小对抗性扰动的影响。即使经过对抗训练，它们在受扰动输入上的性能（鲁棒泛化）也远低于其在良性输入上的最优可达性能（标准泛化）。因此，理解过参数化如何从根本上影响鲁棒性至关重要。本文聚焦于随机特征回归模型（具有随机第一层权重的两层神经网络），精确刻画过参数化在鲁棒性中的作用。我们考虑样本量、输入维度和参数数量按比例增长的机制，并推导出模型经过对抗训练时鲁棒泛化误差的渐近精确公式。我们发展的理论揭示了过参数化对鲁棒性的非平凡影响，表明对于经过对抗训练的随机特征模型，高度过参数化可能会损害鲁棒泛化性能。