Recent work on algorithmic fairness has largely focused on the fairness of discrete decisions, or classifications. While such decisions are often based on risk score models, the fairness of the risk models themselves has received considerably less attention. Risk models are of interest for a number of reasons, including the fact that they communicate uncertainty about the potential outcomes to users, thus representing a way to enable meaningful human oversight. Here, we address fairness desiderata for risk score models. We identify the provision of similar epistemic value to different groups as a key desideratum for risk score fairness. Further, we address how to assess the fairness of risk score models quantitatively, including a discussion of metric choices and meaningful statistical comparisons between groups. In this context, we also introduce a novel calibration error metric that is less sample size-biased than previously proposed metrics, enabling meaningful comparisons between groups of different sizes. We illustrate our methodology - which is widely applicable in many other settings - in two case studies, one in recidivism risk prediction, and one in risk of major depressive disorder (MDD) prediction.

翻译：近期关于算法公平性的工作主要集中于离散决策或分类的公平性。尽管此类决策常基于风险评分模型，但风险模型本身的公平性所受关注却相对不足。风险模型之所以值得研究，原因诸多，包括其能向用户传达关于潜在结果的不确定性，从而为实现有意义的监督提供途径。本文针对风险评分模型的公平性需求展开讨论，将向不同群体提供相似认知价值确立为风险评分公平性的关键要求。此外，本文探讨了如何定量评估风险评分模型的公平性，包括指标选择以及群体间有意义的统计比较。在此背景下，我们引入了一种新颖的校准误差指标，该指标相比此前提出的指标受样本量偏差影响更小，从而实现不同规模群体间的合理比较。我们通过两个案例研究（累犯风险预测与重度抑郁障碍风险预测）展示了这一具有广泛适用性的方法论。