Despite the empirical advances of deep learning across a variety of learning tasks, our theoretical understanding of its success is still very restricted. One of the key challenges is the overparametrized nature of modern models, enabling complete overfitting of the data even if the labels are randomized, i.e. networks can completely \textit{memorize} all given patterns. While such a memorization capacity seems worrisome, in this work we show that under training protocols that include \textit{data augmentation}, neural networks learn to memorize entirely random labels in a benign way, i.e. they learn embeddings that lead to highly non-trivial performance under nearest neighbour probing. We demonstrate that deep models have the surprising ability to separate noise from signal by distributing the task of memorization and feature learning to different layers. As a result, only the very last layers are used for memorization, while preceding layers encode performant features which remain largely unaffected by the label noise. We explore the intricate role of the augmentations used for training and identify a memorization-generalization trade-off in terms of their diversity, marking a clear distinction to all previous works. Finally, we give a first explanation for the emergence of benign memorization by showing that \textit{malign} memorization under data augmentation is infeasible due to the insufficient capacity of the model for the increased sample size. As a consequence, the network is forced to leverage the correlated nature of the augmentations and as a result learns meaningful features. To complete the picture, a better theory of feature learning in deep neural networks is required to fully understand the origins of this phenomenon.

翻译：尽管深度学习在各类学习任务上取得了实证进展，但对其成功背后的理论理解仍然极其有限。现代模型过度参数化的特性是核心挑战之一，这使得模型即使在标签被随机化时也能完全过拟合数据——即网络能够彻底"记忆"所有给定的模式。虽然这种记忆化能力看似令人担忧，但本研究表明，在包含数据增强的训练协议下，神经网络会以良性方式学习记忆完全随机的标签：它们学到的嵌入能够在最近邻探针测试中展现出高度非平凡的性能。我们证明深度模型具有将记忆化任务与特征学习任务分配到不同层次以分离噪声与信号的惊人能力。最终，仅最后几层用于记忆化，而前层编码的性能特征几乎不受标签噪声影响。我们探讨了训练所用数据增强的复杂作用，并基于其多样性识别出记忆化-泛化权衡关系，这与所有先前研究形成鲜明对比。最后，我们首次对良性记忆化的出现给出解释：由于模型对增广样本规模的容量不足，数据增强条件下的"恶性"记忆化不可行。因此，网络被迫利用增强数据的相关性质，从而学习到有意义的特征。要完善这一图景，需要更深入的深度神经网络特征学习理论来全面理解该现象的起源。