Quantifying the impact of training data points is crucial for understanding the outputs of machine learning models and for improving the transparency of the AI pipeline. The influence function is a principled and popular data attribution method, but its computational cost often makes it challenging to use. This issue becomes more pronounced in the setting of large language models and text-to-image models. In this work, we propose DataInf, an efficient influence approximation method that is practical for large-scale generative AI models. Leveraging an easy-to-compute closed-form expression, DataInf outperforms existing influence computation algorithms in terms of computational and memory efficiency. Our theoretical analysis shows that DataInf is particularly well-suited for parameter-efficient fine-tuning techniques such as LoRA. Through systematic empirical evaluations, we show that DataInf accurately approximates influence scores and is orders of magnitude faster than existing methods. In applications to RoBERTa-large, Llama-2-13B-chat, and stable-diffusion-v1.5 models, DataInf effectively identifies the most influential fine-tuning examples better than other approximate influence scores. Moreover, it can help to identify which data points are mislabeled.

翻译：量化训练数据点的影响对于理解机器学习模型的输出及提升AI管线的透明度至关重要。影响函数是一种基于原理且广泛使用的数据归因方法，但其计算成本往往使其难以应用。这一问题在大语言模型和文本到图像模型中尤为突出。本文提出DataInf，一种适用于大规模生成式AI模型的高效影响力近似方法。通过利用易于计算的闭式表达式，DataInf在计算效率和内存效率上均优于现有影响力计算算法。理论分析表明，DataInf特别适用于参数高效微调技术（如LoRA）。通过系统性的实证评估，DataInf能够精确近似影响力分数，且速度比现有方法快数个数量级。在RoBERTa-large、Llama-2-13B-chat以及stable-diffusion-v1.5模型的应用中，DataInf比其它近似影响力分数更有效地识别出最具影响力的微调样本。此外，它还能帮助识别哪些数据点存在标注错误。