Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution -- quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that can be decomposed into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to $10^3$ times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.

翻译：影响函数作为一种源于稳健统计学的技术，在现代机器学习中被应用于一项新颖的任务：数据归因——量化单个训练数据点如何影响模型的预测。然而，数据归因文献中常见的影响函数推导仅限于可分解为单个数据点损失之和的损失函数，其中最著名的例子是M估计量。这限制了影响函数在更复杂学习目标（我们称之为非可分解损失）上的应用，例如对比损失或排序损失，其中单个损失项依赖于多个数据点且无法进一步分解。在本工作中，我们通过重新审视稳健统计学中影响函数的一般形式（其适用范围超越了M估计量）来弥合这一差距。基于此形式，我们提出了一种新方法——通用影响函数（VIF），该方法可直接应用于使用任何非可分解损失训练的机器学习模型。与统计学中的经典方法相比，所提出的VIF旨在充分利用自动微分的能力，从而无需针对每种损失函数进行特定推导。我们在三个示例中验证了VIF的有效性：用于生存分析的Cox回归、用于网络分析的节点嵌入以及用于信息检索的列表式学习排序。在所有案例中，VIF估计的影响值与通过暴力留一法重新训练得到的结果高度吻合，同时计算速度提升了高达$10^3$倍。我们相信VIF代表了数据归因领域的重要进展，能够在广泛的机器学习范式中实现基于影响函数的高效归因，具有广阔的实际应用潜力。