Data valuation has wide use cases in machine learning, including improving data quality and creating economic incentives for data sharing. This paper studies the robustness of data valuation to noisy model performance scores. Particularly, we find that the inherent randomness of the widely used stochastic gradient descent can cause existing data value notions (e.g., the Shapley value and the Leave-one-out error) to produce inconsistent data value rankings across different runs. To address this challenge, we introduce the concept of safety margin, which measures the robustness of a data value notion. We show that the Banzhaf value, a famous value notion that originated from cooperative game theory literature, achieves the largest safety margin among all semivalues (a class of value notions that satisfy crucial properties entailed by ML applications and include the famous Shapley value and Leave-one-out error). We propose an algorithm to efficiently estimate the Banzhaf value based on the Maximum Sample Reuse (MSR) principle. Our evaluation demonstrates that the Banzhaf value outperforms the existing semivalue-based data value notions on several ML tasks such as learning with weighted samples and noisy label detection. Overall, our study suggests that when the underlying ML algorithm is stochastic, the Banzhaf value is a promising alternative to the other semivalue-based data value schemes given its computational advantage and ability to robustly differentiate data quality.

翻译：数据估值在机器学习中具有广泛的应用，包括提升数据质量和为数据共享创造经济激励。本文研究了数据估值对含噪声模型性能得分的鲁棒性。具体而言，我们发现广泛使用的随机梯度下降的固有随机性会导致现有数据价值概念（如Shapley值和留一法误差）在不同运行中产生不一致的数据价值排名。为解决这一挑战，我们引入了安全裕度的概念，用于衡量数据价值概念的鲁棒性。我们证明了源于合作博弈论文献的著名价值概念——Banzhaf值，在所有半值（一类满足机器学习应用关键性质并包含著名Shapley值和留一法误差的价值概念）中实现了最大的安全裕度。我们提出了一种基于最大样本复用原则的高效Banzhaf值估计算法。实验评估表明，在加权样本学习和噪声标签检测等若干机器学习任务中，Banzhaf值优于现有的基于半值的数据价值概念。总体而言，本研究建议，当底层机器学习算法具有随机性时，Banzhaf值因其计算优势和对数据质量鲁棒性区分的能力，是其他基于半值的数据价值方案的有前景替代方案。