The proliferation of large-scale AI models trained on extensive datasets has revolutionized machine learning. With these models taking on increasingly central roles in various applications, the need to understand their behavior and enhance interpretability has become paramount. To investigate the impact of changes in training data on a pre-trained model, a common approach is leave-one-out retraining. This entails systematically altering the training dataset by removing specific samples to observe resulting changes within the model. However, retraining the model for each altered dataset presents a significant computational challenge, given the need to perform this operation for every dataset variation. In this paper, we introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. During the offline training phase, we approximate the influence of training data on the target model through a distilled synset, formulated as a reversed gradient matching problem. For online evaluation, we expedite the leave-one-out process using the synset, which is then utilized to compute the attribution matrix based on the evaluation objective. Experimental evaluations, including training data attribution and assessments of data quality, demonstrate that our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.

翻译：大规模AI模型在广泛数据集上的训练激增，彻底改变了机器学习。随着这些模型在各类应用中扮演越来越核心的角色，理解其行为并增强可解释性变得至关重要。为探究训练数据变化对预训练模型的影响，一种常见方法是留一法再训练，即系统性地修改训练数据集（移除特定样本）以观察模型中的相应变化。然而，针对每种修改后的数据集进行模型再训练存在显著的计算挑战，因为需要对每种数据变体执行该操作。本文提出了一种高效的数据影响评估框架，包含离线训练和在线评估阶段。在离线训练阶段，我们通过蒸馏合成集（distilled synset）来近似训练数据对目标模型的影响，并将其建模为一个逆梯度匹配问题。在在线评估中，我们利用该合成集加速留一法过程，进而基于评估目标计算归因矩阵。实验评估（包括训练数据归因和数据质量评估）表明，与直接再训练方法相比，本方法能在显著加速的同时实现可比的模型行为评估。