Overfitting negatively impacts the generalization ability of deep neural networks (DNNs) in both natural and adversarial training. Existing methods struggle to consistently address different types of overfitting, typically designing strategies that focus separately on either natural or adversarial patterns. In this work, we adopt a unified perspective by solely focusing on natural patterns to explore different types of overfitting. Specifically, we examine the memorization effect in DNNs and reveal a shared behaviour termed over-memorization, which impairs their generalization capacity. This behaviour manifests as DNNs suddenly becoming high-confidence in predicting certain training patterns and retaining a persistent memory for them. Furthermore, when DNNs over-memorize an adversarial pattern, they tend to simultaneously exhibit high-confidence prediction for the corresponding natural pattern. These findings motivate us to holistically mitigate different types of overfitting by hindering the DNNs from over-memorization natural patterns. To this end, we propose a general framework, Distraction Over-Memorization (DOM), which explicitly prevents over-memorization by either removing or augmenting the high-confidence natural patterns. Extensive experiments demonstrate the effectiveness of our proposed method in mitigating overfitting across various training paradigms.

翻译：过拟合在自然训练与对抗训练中均严重损害深度神经网络（DNNs）的泛化能力。现有方法难以一致性地处理不同类型的过拟合，通常需要为自然模式和对抗模式分别设计针对性策略。本研究从统一视角出发，仅聚焦于自然模式来探索不同过拟合类型。具体而言，我们考察了DNNs中的记忆效应，揭示了一种称为"过记忆"的共性行为——该行为会损害模型的泛化能力，表现为DNNs突然对特定训练模式产生高置信度预测，并持续保留这些记忆。进一步发现，当DNNs对对抗模式产生过记忆时，往往会同步对相应自然模式表现出高置信度预测。这些发现启发我们通过阻止DNNs对自然模式的过记忆来整体性缓解不同过拟合类型。为此，我们提出一个通用框架——分散过记忆（DOM），通过移除或增强高置信度自然模式来显式防止过记忆。大量实验表明，本方法在多种训练范式下均能有效缓解过拟合问题。