Data valuation has wide use cases in machine learning, including improving data quality and creating economic incentives for data sharing. This paper studies the robustness of data valuation to noisy model performance scores. Particularly, we find that the inherent randomness of the widely used stochastic gradient descent can cause existing data value notions (e.g., the Shapley value and the Leave-one-out error) to produce inconsistent data value rankings across different runs. To address this challenge, we introduce the concept of safety margin, which measures the robustness of a data value notion. We show that the Banzhaf value, a famous value notion that originated from cooperative game theory literature, achieves the largest safety margin among all semivalues (a class of value notions that satisfy crucial properties entailed by ML applications and include the famous Shapley value and Leave-one-out error). We propose an algorithm to efficiently estimate the Banzhaf value based on the Maximum Sample Reuse (MSR) principle. Our evaluation demonstrates that the Banzhaf value outperforms the existing semivalue-based data value notions on several ML tasks such as learning with weighted samples and noisy label detection. Overall, our study suggests that when the underlying ML algorithm is stochastic, the Banzhaf value is a promising alternative to the other semivalue-based data value schemes given its computational advantage and ability to robustly differentiate data quality.

翻译：数据估值在机器学习中具有广泛的应用场景，包括提升数据质量和为数据共享创造经济激励。本文研究数据估值对噪声模型性能得分的鲁棒性。特别地，我们发现广泛使用的随机梯度下降法固有的随机性会导致现有数据估值概念（如沙普利值和留一法误差）在不同运行中产生不一致的数据价值排序。为解决这一挑战，我们引入安全边际的概念，用以衡量数据估值概念的鲁棒性。我们证明，发源于合作博弈论文献的著名估值概念——班扎夫值——在所有半值（一类满足机器学习应用关键属性且包含著名沙普利值和留一法误差的估值概念）中具有最大的安全边际。我们提出一种基于最大样本复用（MSR）原则的高效班扎夫值估计算法。实验评估表明，在加权样本学习和噪声标签检测等多个机器学习任务中，班扎夫值优于现有的基于半值的数据估值概念。总体而言，本研究表明，当底层机器学习算法具有随机性时，班扎夫值因其计算优势及鲁棒区分数据质量的能力，是其他基于半值的数据估值方案的有前景的替代选择。