As large language models are increasingly trained and fine-tuned, practitioners need methods to identify which training data drive specific behaviors, particularly unintended ones. Training Data Attribution (TDA) methods address this by estimating datapoint influence. Existing approaches like influence functions are both computationally expensive and attribute based on single test examples, which can bias results toward syntactic rather than semantic similarity. To address these issues of scalability and influence to abstract behavior, we leverage interpretable structures within the model during the attribution. First, we introduce Concept Influence which attribute model behavior to semantic directions (such as linear probes or sparse autoencoder features) rather than individual test examples. Second, we show that simple probe-based attribution methods are first-order approximations of Concept Influence that achieve comparable performance while being over an order-of-magnitude faster. We empirically validate Concept Influence and approximations across emergent misalignment benchmarks and real post-training datasets, and demonstrate they achieve comparable performance to classical influence functions while being substantially more scalable. More broadly, we show that incorporating interpretable structure within traditional TDA pipelines can enable more scalable, explainable, and better control of model behavior through data.

翻译：随着大型语言模型日益频繁地进行训练与微调，从业者亟需能够识别驱动特定行为（尤其是非预期行为）的训练数据的方法。训练数据归因方法通过估计数据点的影响力来解决这一问题。现有方法（如影响函数）不仅计算成本高昂，且基于单一测试样本进行归因，这可能导致结果偏向句法相似性而非语义相似性。为应对可扩展性及对抽象行为的影响力归因问题，我们在归因过程中利用模型内部的可解释结构。首先，我们提出概念影响力方法，将模型行为归因于语义方向（例如线性探针或稀疏自编码器特征），而非单个测试样本。其次，我们证明基于探针的简单归因方法是概念影响力的一阶近似，在实现相当性能的同时，计算速度提升超过一个数量级。我们在涌现错位基准测试和实际后训练数据集上对概念影响力及其近似方法进行了实证验证，结果表明它们在保持与经典影响函数相当性能的同时，具备显著更高的可扩展性。更广泛而言，我们证明将可解释结构融入传统训练数据归因流程，能够通过数据实现更具可扩展性、可解释性且更优的模型行为控制。