Training data attribution (TDA) methods aim to quantify the influence of individual training data points on the model predictions, with broad applications in data-centric AI, such as mislabel detection, data selection, and copyright compensation. However, existing methods in this field, which can be categorized as retraining-based and gradient-based, have struggled with the trade-off between computational efficiency and attribution efficacy. Retraining-based methods can accurately attribute complex non-convex models but are computationally prohibitive, while gradient-based methods are efficient but often fail for non-convex models. Recent research has shown that augmenting gradient-based methods with ensembles of multiple independently trained models can achieve significantly better attribution efficacy. However, this approach remains impractical for very large-scale applications. In this work, we discover that expensive, fully independent training is unnecessary for ensembling the gradient-based methods, and we propose two efficient ensemble strategies, DROPOUT ENSEMBLE and LORA ENSEMBLE, alternative to naive independent ensemble. These strategies significantly reduce training time (up to 80%), serving time (up to 60%), and space cost (up to 80%) while maintaining similar attribution efficacy to the naive independent ensemble. Our extensive experimental results demonstrate that the proposed strategies are effective across multiple TDA methods on diverse datasets and models, including generative settings, significantly advancing the Pareto frontier of TDA methods with better computational efficiency and attribution efficacy.

翻译：训练数据归因（TDA）方法旨在量化单个训练数据点对模型预测的影响，在数据为中心的AI领域具有广泛应用，例如错误标签检测、数据选择和版权补偿。然而，该领域现有方法可分为基于重训练和基于梯度的两类，始终面临计算效率与归因效能之间的权衡难题。基于重训练的方法能够准确归因复杂的非凸模型，但计算成本过高；而基于梯度的方法虽计算高效，却常对非凸模型失效。近期研究表明，通过集成多个独立训练模型来增强基于梯度的方法，可显著提升归因效能。然而，该方法在超大规模应用中仍不具实用性。本研究发现，对于基于梯度方法的集成，昂贵且完全独立的训练并非必要，我们提出了两种高效集成策略——DROPOUT ENSEMBLE与LORA ENSEMBLE，以替代朴素的独立集成。这些策略在保持与朴素独立集成相近归因效能的同时，显著降低了训练时间（最高80%）、服务时间（最高60%）和空间成本（最高80%）。大量实验结果表明，所提策略在多种数据集和模型（包括生成式场景）上对多种TDA方法均有效，通过更优的计算效率与归因效能显著推进了TDA方法的帕累托前沿。