Deep learning models are known to overfit and memorize spurious features in the training dataset. While numerous empirical studies have aimed at understanding this phenomenon, a rigorous theoretical framework to quantify it is still missing. In this paper, we consider spurious features that are uncorrelated with the learning task, and we provide a precise characterization of how they are memorized via two separate terms: (i) the stability of the model with respect to individual training samples, and (ii) the feature alignment between the spurious feature and the full sample. While the first term is well established in learning theory and it is connected to the generalization error in classical work, the second one is, to the best of our knowledge, novel. Our key technical result gives a precise characterization of the feature alignment for the two prototypical settings of random features (RF) and neural tangent kernel (NTK) regression. We prove that the memorization of spurious features weakens as the generalization capability increases and, through the analysis of the feature alignment, we unveil the role of the model and of its activation function. Numerical experiments show the predictive power of our theory on standard datasets (MNIST, CIFAR-10).

翻译：深度学习模型已知会过拟合并记忆训练数据集中的伪特征。尽管大量实证研究旨在理解这一现象，但目前仍缺乏严谨的理论框架对其进行量化。本文考虑与学习任务无关的伪特征，并通过两个独立项精确刻画其记忆机制：（i）模型对单个训练样本的稳定性；（ii）伪特征与完整样本之间的特征对齐度。第一项在学习理论中已得到充分确立，并与经典工作中的泛化误差相关联，而据我们所知，第二项是全新的。我们的关键技术结果为随机特征（RF）与神经正切核（NTK）回归这两个典型设定提供了特征对齐度的精确刻画。我们证明，伪特征的记忆会随着泛化能力的增强而减弱，并通过特征对齐度分析揭示了模型及其激活函数的作用。数值实验在标准数据集（MNIST、CIFAR-10）上展示了我们理论的预测能力。