In this paper, we propose a novel data-pruning approach called moving-one-sample-out (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings.

翻译：本文提出一种名为“通过移除单样本实现数据剪枝”（MoSo）的新型数据剪枝方法，旨在识别并从训练集中移除信息量最低的样本。MoSo的核心思想是通过评估每个样本对最优经验风险的影响来确定其重要性，具体通过测量从训练集中剔除特定样本时经验风险的变化程度实现。为避免计算代价高昂的留一重训练过程，我们提出一种仅需不同训练阶段梯度信息的高效一阶近似方法。该近似的核心思想在于：梯度方向与训练集平均梯度持续一致的样本更具信息量，应获得更高评分。其直观理解如下：若某样本的梯度与平均梯度向量一致，则利用该样本优化网络将对所有剩余样本产生类似效果。实验结果表明，MoSo在高剪枝率下有效缓解了严重性能退化问题，并在多种设定下均取得令人满意的表现。