We empirically investigate the impact of learning randomly generated labels in parallel to class labels in supervised learning on memorization, model complexity, and generalization in deep neural networks. To this end, we introduce a multi-head network architecture as an extension of standard CNN architectures. Inspired by methods used in fair AI, our approach allows for the unlearning of random labels, preventing the network from memorizing individual samples. Based on the concept of Rademacher complexity, we first use our proposed method as a complexity metric to analyze the effects of common regularization techniques and challenge the traditional understanding of feature extraction and classification in CNNs. Second, we propose a novel regularizer that effectively reduces sample memorization. However, contrary to the predictions of classical statistical learning theory, we do not observe improvements in generalization.

翻译：我们通过实证研究在监督学习中并行学习随机生成标签与类别标签对深度神经网络的记忆化、模型复杂度和泛化能力的影响。为此，我们引入一种多头网络架构作为标准CNN架构的扩展。受公平AI领域方法的启发，我们的方法能够实现随机标签的"遗忘"，防止网络记忆个体样本。基于Rademacher复杂度的概念，我们首先将所提方法作为复杂度度量，分析常见正则化技术的影响，并对CNN中特征提取与分类的传统理解提出挑战。其次，我们提出一种新型正则化器，能有效减少样本记忆化。然而，与经典统计学习理论的预测相反，我们并未观察到泛化性能的改善。