Machine learning models are vulnerable to adversarial perturbations, and a thought-provoking paper by Bubeck and Sellke has analyzed this phenomenon through the lens of over-parameterization: interpolating smoothly the data requires significantly more parameters than simply memorizing it. However, this "universal" law provides only a necessary condition for robustness, and it is unable to discriminate between models. In this paper, we address these gaps by focusing on empirical risk minimization in two prototypical settings, namely, random features and the neural tangent kernel (NTK). We prove that, for random features, the model is not robust for any degree of over-parameterization, even when the necessary condition coming from the universal law of robustness is satisfied. In contrast, for even activations, the NTK model meets the universal lower bound, and it is robust as soon as the necessary condition on over-parameterization is fulfilled. This also addresses a conjecture in prior work by Bubeck, Li and Nagaraj. Our analysis decouples the effect of the kernel of the model from an "interaction matrix", which describes the interaction with the test data and captures the effect of the activation. Our theoretical results are corroborated by numerical evidence on both synthetic and standard datasets (MNIST, CIFAR-10).

翻译：机器学习模型对对抗性扰动敏感，Bubeck和Sellke一篇发人深省的论文通过过参数化的视角分析了这一现象：平滑插值数据所需的参数远多于简单记忆数据。然而，这一“普适”定律仅提供了鲁棒性的必要条件，且无法区分不同模型。本文通过关注两个典型设定下的经验风险最小化（即随机特征与神经正切核）来填补这些空白。我们证明：对于随机特征，无论过参数化程度如何，模型均不鲁棒，即使满足普适鲁棒定律的必要条件；相反，对于偶数激活函数，神经正切核模型满足普适下界，且一旦满足过参数化的必要条件即具有鲁棒性。这同时解决了Bubeck、Li和Nagaraj此前工作中的一项猜想。我们的分析将模型核函数的影响与“交互矩阵”解耦，后者描述与测试数据的交互并捕捉激活函数的影响。理论结果在合成数据集和标准数据集（MNIST、CIFAR-10）上的数值实验中得到验证。