Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor generalization when the learned explanations rely on spurious correlations. In this work, we formalize the interplay between memorization and generalization, showing that spurious correlations would particularly lead to poor generalization when are combined with memorization. Memorization can reduce training loss to zero, leaving no incentive to learn robust, generalizable patterns. To address this, we propose memorization-aware training (MAT), which uses held-out predictions as a signal of memorization to shift a model's logits. MAT encourages learning robust patterns invariant across distributions, improving generalization under distribution shifts.

翻译：神经网络通常学习能够拟合大多数数据的简单解释，同时记忆偏离这些解释的异常情况。当学习到的解释依赖于虚假相关性时，这种行为会导致较差的泛化能力。在这项工作中，我们形式化了记忆与泛化之间的相互作用，表明虚假相关性尤其在与记忆结合时会导致较差的泛化。记忆可以将训练损失降至零，从而失去学习鲁棒、可泛化模式的动力。为了解决这个问题，我们提出了记忆感知训练（MAT），它使用留出预测作为记忆信号来调整模型的逻辑输出。MAT鼓励学习跨分布不变的鲁棒模式，从而改善分布偏移下的泛化能力。