Demographic parity is the most widely recognized measure of group fairness in machine learning, which ensures equal treatment of different demographic groups. Numerous works aim to achieve demographic parity by pursuing the commonly used metric $\Delta DP$. Unfortunately, in this paper, we reveal that the fairness metric $\Delta DP$ can not precisely measure the violation of demographic parity, because it inherently has the following drawbacks: i) zero-value $\Delta DP$ does not guarantee zero violation of demographic parity, ii) $\Delta DP$ values can vary with different classification thresholds. To this end, we propose two new fairness metrics, Area Between Probability density function Curves (ABPC) and Area Between Cumulative density function Curves (ABCC), to precisely measure the violation of demographic parity at the distribution level. The new fairness metrics directly measure the difference between the distributions of the prediction probability for different demographic groups. Thus our proposed new metrics enjoy: i) zero-value ABCC/ABPC guarantees zero violation of demographic parity; ii) ABCC/ABPC guarantees demographic parity while the classification thresholds are adjusted. We further re-evaluate the existing fair models with our proposed fairness metrics and observe different fairness behaviors of those models under the new metrics. The code is available at https://github.com/ahxt/new_metric_for_demographic_parity

翻译：群体平等（Demographic parity）是机器学习中最广泛认可的群体公平性度量标准，旨在确保不同人口群体受到平等对待。大量研究致力于通过使用常用指标 $\Delta DP$ 来实现群体平等。然而，本文揭示了公平性指标 $\Delta DP$ 无法精确衡量群体平等的违反程度，因其固有缺陷包括：i) 零值的 $\Delta DP$ 并不保证群体平等完全不被违反，ii) $\Delta DP$ 的值会随分类阈值的变化而改变。为此，我们提出了两个新的公平性指标——概率密度曲线间面积（Area Between Probability density function Curves, ABPC）和累积分布曲线间面积（Area Between Cumulative density function Curves, ABCC），以在分布层面精确衡量群体平等的违反程度。这两个新指标直接度量不同群体预测概率分布之间的差异。因此，我们提出的新指标具备以下优势：i) 零值的 ABCC/ABPC 保证群体平等不被违反；ii) ABCC/ABPC 在调整分类阈值时仍能保证群体平等。我们进一步使用所提出的公平性指标重新评估现有公平模型，并观察到这些模型在新指标下呈现不同的公平行为。代码开源在 https://github.com/ahxt/new_metric_for_demographic_parity