Deep neural networks are over-parameterized and easily overfit the datasets they train on. In the extreme case, it has been shown that these networks can memorize a training set with fully randomized labels. We propose using the curvature of loss function around each training sample, averaged over training epochs, as a measure of memorization of the sample. We use this metric to study the generalization versus memorization properties of different samples in popular image datasets and show that it captures memorization statistics well, both qualitatively and quantitatively. We first show that the high curvature samples visually correspond to long-tailed, mislabeled, or conflicting samples, those that are most likely to be memorized. This analysis helps us find, to the best of our knowledge, a novel failure mode on the CIFAR100 and ImageNet datasets: that of duplicated images with differing labels. Quantitatively, we corroborate the validity of our scores via two methods. First, we validate our scores against an independent and comprehensively calculated baseline, by showing high cosine similarity with the memorization scores released by Feldman and Zhang (2020). Second, we inject corrupted samples which are memorized by the network, and show that these are learned with high curvature. To this end, we synthetically mislabel a random subset of the dataset. We overfit a network to it and show that sorting by curvature yields high AUROC values for identifying the corrupted samples. An added advantage of our method is that it is scalable, as it requires training only a single network as opposed to the thousands trained by the baseline, while capturing the aforementioned failure mode that the baseline fails to identify.

翻译：深度神经网络是过参数化的，容易对其训练数据集产生过拟合。极端情况下，已有研究表明这些网络能够记忆完全随机标注的训练集。我们提出使用每个训练样本周围损失函数的曲率（在训练轮次上取平均）作为该样本记忆化的度量。利用这一指标，我们研究了主流图像数据集中不同样本的泛化与记忆化特性，并从定性和定量两个层面证明该指标能有效捕捉记忆化统计特征。首先，我们发现高曲率样本在视觉上对应于长尾、误标或冲突样本，这些样本最易被网络记忆。借助这一分析，我们发现了CIFAR100和ImageNet数据集上一类新的（据我们所知）故障模式：存在标签不同的重复图像。在定量验证方面，我们通过两种方法证实了评分的有效性。其一，通过显示与Feldman和Zhang (2020) 发布的记忆化分数具有高余弦相似度，我们将评分结果与独立且全面计算的基线进行对比验证。其二，我们向数据集中注入被网络记忆的污染样本，并证明这些样本在学习过程中呈现高曲率特征。具体而言，我们对数据集随机子集进行人为误标，让网络对该数据集过拟合，随后发现按曲率排序能够以高AUROC值识别出污染样本。本方法的另一优势在于其可扩展性：仅需训练单一网络（而基线方法需训练数千个网络），就能捕捉到基线方法无法识别的上述故障模式。