A core data-centric learning challenge is the identification of training samples that are detrimental to model performance. Influence functions serve as a prominent tool for this task and offer a robust framework for assessing training data influence on model predictions. Despite their widespread use, their high computational cost associated with calculating the inverse of the Hessian matrix pose constraints, particularly when analyzing large-sized deep models. In this paper, we establish a bridge between identifying detrimental training samples via influence functions and outlier gradient detection. This transformation not only presents a straightforward and Hessian-free formulation but also provides insights into the role of the gradient in sample impact. Through systematic empirical evaluations, we first validate the hypothesis of our proposed outlier gradient analysis approach on synthetic datasets. We then demonstrate its effectiveness in detecting mislabeled samples in vision models and selecting data samples for improving performance of natural language processing transformer models. We also extend its use to influential sample identification for fine-tuning Large Language Models.

翻译：以数据为中心的学习面临一个核心挑战：如何识别对模型性能有害的训练样本。影响函数作为解决该任务的重要工具，为评估训练数据对模型预测的影响提供了稳健框架。尽管应用广泛，但其计算海森矩阵逆矩阵所需的高昂计算成本构成了限制，特别是在分析大规模深度模型时。本文在通过影响函数识别有害训练样本与离群梯度检测之间建立了桥梁。该转换不仅提出了简洁且无需海森矩阵的公式化方法，还深入揭示了梯度在样本影响中的作用机制。通过系统性的实证评估，我们首先在合成数据集上验证了所提出的离群梯度分析方法的假设。随后证明了该方法在视觉模型中检测错误标注样本、以及为提升自然语言处理Transformer模型性能而筛选数据样本方面的有效性。我们还将其扩展应用于大语言模型微调过程中的关键样本识别任务。