Demographic parity is the most widely recognized measure of group fairness in machine learning, which ensures equal treatment of different demographic groups. Numerous works aim to achieve demographic parity by pursuing the commonly used metric $\Delta DP$. Unfortunately, in this paper, we reveal that the fairness metric $\Delta DP$ can not precisely measure the violation of demographic parity, because it inherently has the following drawbacks: i) zero-value $\Delta DP$ does not guarantee zero violation of demographic parity, ii) $\Delta DP$ values can vary with different classification thresholds. To this end, we propose two new fairness metrics, Area Between Probability density function Curves (ABPC) and Area Between Cumulative density function Curves (ABCC), to precisely measure the violation of demographic parity at the distribution level. The new fairness metrics directly measure the difference between the distributions of the prediction probability for different demographic groups. Thus our proposed new metrics enjoy: i) zero-value ABCC/ABPC guarantees zero violation of demographic parity; ii) ABCC/ABPC guarantees demographic parity while the classification thresholds are adjusted. We further re-evaluate the existing fair models with our proposed fairness metrics and observe different fairness behaviors of those models under the new metrics. The code is available at https://github.com/ahxt/new_metric_for_demographic_parity

翻译：人口统计均等性是机器学习中最广泛认可的群体公平性度量，旨在确保不同人口群体受到平等对待。众多研究致力于通过广泛使用的度量指标 $\Delta DP$ 来实现人口统计均等性。然而，本文揭示出公平性度量 $\Delta DP$ 无法精确衡量人口统计均等性的违反情况，因为它存在以下固有缺陷：i) 零值的 $\Delta DP$ 并不能保证人口统计均等性完全未受违反，ii) $\Delta DP$ 的值会随分类阈值的变化而改变。为此，我们提出了两种新型公平性度量——概率密度函数曲线间面积（ABPC）与累积分布函数曲线间面积（ABCC），以在分布层面精确衡量人口统计均等性的违反程度。这些新型公平性度量直接度量不同人口群体预测概率分布之间的差异。因此，我们提出的新度量具备以下优势：i) 零值 ABCC/ABPC 可保证人口统计均等性完全未受违反；ii) 在调整分类阈值时，ABCC/ABPC 仍能保证人口统计均等性。我们进一步利用所提出的公平性度量重新评估了现有公平性模型，并观察到这些模型在新度量下呈现出不同的公平性行为。相关代码已开源至 https://github.com/ahxt/new_metric_for_demographic_parity。