Recently, large language and vision models have shown strong performance, but due to high pre-training and fine-tuning costs, research has shifted towards faster training via dataset pruning. Previous methods used sample loss as an evaluation criterion, aiming to select the most "difficult" samples for training. However, when the pruning rate increases, the number of times each sample is trained becomes more evenly distributed, which causes many critical or general samples to not be effectively fitted. We refer to this as Low-Frequency Learning (LFL). In other words, LFL prevents the model from remembering most samples. In our work, we decompose the scoring function of LFL, provide a theoretical explanation for the inefficiency of LFL, and propose adding a memory term to the scoring function to enhance the model's memory capability, along with an approximation of this memory term. Similarly, we explore memory in Self-Supervised Learning (SSL), marking the first discussion on SSL memory. Using contrastive learning, we derive the memory term both theoretically and experimentally. Finally, we propose Enhance Memory Pruning (EMP), which addresses the issue of insufficient memory under high pruning rates by enhancing the model's memory of data, thereby improving its performance. We evaluated the performance of EMP in tasks such as image classification, natural language understanding, and model pre-training. The results show that EMP can improve model performance under extreme pruning rates. For example, in the CIFAR100-ResNet50 pre-training task, with 70\% pruning, EMP outperforms current methods by 2.2\%.

翻译：近年来，大型语言与视觉模型展现出强大性能，但由于预训练与微调成本高昂，研究重心逐渐转向通过数据集剪枝实现快速训练。现有方法通常以样本损失作为评估标准，旨在选取最“困难”的样本进行训练。然而，当剪枝率提高时，每个样本被训练的频次分布趋于均匀，导致许多关键性或通用性样本无法得到有效拟合。我们将此现象称为低频学习（LFL）。换言之，LFL阻碍了模型对多数样本的记忆能力。在本研究中，我们解析了LFL评分函数的构成，从理论上阐释了LFL效率低下的原因，并提出在评分函数中引入记忆项以增强模型记忆能力，同时给出了该记忆项的近似计算方法。进一步地，我们探讨了自监督学习（SSL）中的记忆机制，首次系统论述了SSL环境下的记忆问题。基于对比学习框架，我们通过理论推导与实验验证获得了记忆项的具体形式。最终，我们提出增强记忆剪枝（EMP）方法，通过强化模型对数据的记忆能力，解决高剪枝率下记忆不足的问题，从而提升模型性能。我们在图像分类、自然语言理解及模型预训练等任务中评估了EMP的性能。实验结果表明，EMP能够在极端剪枝率下有效提升模型表现。例如在CIFAR100-ResNet50预训练任务中，70%剪枝率下EMP相较现有方法实现了2.2%的性能提升。