Measures of algorithmic fairness are usually discussed in the context of binary decisions. We extend the approach to continuous scores. So far, ROC-based measures have mainly been suggested for this purpose. Other existing methods depend heavily on the distribution of scores, are unsuitable for ranking tasks, or their effect sizes are not interpretable. Here, we propose a distributionally invariant version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and interpreting the strength of group disparities as well as for comparing biases across different models, datasets, or time points. We derive a link between the different families of existing fairness measures for scores and show that the proposed distributionally invariant fairness measures outperform ROC-based fairness measures because they are more explicit and can quantify significant biases that ROC-based fairness measures miss. Finally, we demonstrate their effectiveness through experiments on the most commonly used fairness benchmark datasets.

翻译：算法公平性的度量通常围绕二分类决策进行讨论。我们将该方法扩展到连续得分场景。目前，基于ROC的度量主要被用于此目的，但其他现有方法严重依赖得分的分布，不适用于排序任务，或效应量缺乏可解释性。本文提出一种基于Wasserstein距离的分布不变公平性度量方法，具有合理的解释性。该度量易于计算，适用于量化与解释群体差异的强度，以及比较不同模型、数据集或时间点间的偏差。我们推导了现有连续得分公平性度量不同家族间的关联，并表明所提出的分布不变公平性度量优于基于ROC的公平性度量，因其更明确且能量化后者遗漏的显著偏差。最后，通过在常用公平性基准数据集上的实验验证了其有效性。