Recent work on algorithmic fairness has largely focused on the fairness of discrete decisions, or classifications. While such decisions are often based on risk score models, the fairness of the risk models themselves has received considerably less attention. Risk models are of interest for a number of reasons, including the fact that they communicate uncertainty about the potential outcomes to users, thus representing a way to enable meaningful human oversight. Here, we address fairness desiderata for risk score models. We identify the provision of similar epistemic value to different groups as a key desideratum for risk score fairness. Further, we address how to assess the fairness of risk score models quantitatively, including a discussion of metric choices and meaningful statistical comparisons between groups. In this context, we also introduce a novel calibration error metric that is less sample size-biased than previously proposed metrics, enabling meaningful comparisons between groups of different sizes. We illustrate our methodology - which is widely applicable in many other settings - in two case studies, one in recidivism risk prediction, and one in risk of major depressive disorder (MDD) prediction.

翻译：关于算法公平性的近期研究主要聚焦于离散决策或分类的公平性。尽管此类决策通常基于风险评分模型，但风险模型本身的公平性受到的关注则显著较少。风险模型之所以值得关注，原因众多，包括其将潜在结果的不确定性传达给用户，从而为赋予人类有意义的监督提供了一种途径。在此，我们探讨了风险评分模型的公平性准则。我们将为不同群体提供相似的认知价值确定为风险评分公平性的关键准则。此外，我们讨论了如何定量评估风险评分模型的公平性，包括指标选择以及群体间有意义的统计比较的讨论。在此背景下，我们还引入了一种新颖的校准误差度量，该度量相较于先前提出的指标受样本量偏差影响更小，从而使得不同规模群体间的比较具有意义。我们通过两个案例研究——一个关于累犯风险预测，另一个关于重度抑郁症（MDD）风险预测——来阐述我们的方法论，该方法论广泛适用于其他众多场景。