While methods for measuring and correcting differential performance in risk prediction models have proliferated in recent years, most existing techniques can only be used to assess fairness across relatively large subgroups. The purpose of algorithmic fairness efforts is often to redress discrimination against groups that are both marginalized and small, so this sample size limitation often prevents existing techniques from accomplishing their main aim. We take a three-pronged approach to address the problem of quantifying fairness with small subgroups. First, we propose new estimands built on the "counterfactual fairness" framework that leverage information across groups. Second, we estimate these quantities using a larger volume of data than existing techniques. Finally, we propose a novel data borrowing approach to incorporate "external data" that lacks outcomes and predictions but contains covariate and group membership information. This less stringent requirement on the external data allows for more possibilities for external data sources. We demonstrate practical application of our estimators to a risk prediction model used by a major Midwestern health system during the COVID-19 pandemic.

翻译：近年来，尽管衡量和纠正风险预测模型中差异性表现的方法层出不穷，但现有技术大多仅能评估规模相对较大的群体公平性。算法公平性工作的核心目标往往是消除对边缘化且规模较小群体的歧视性影响，因此样本量的限制常使现有技术难以达成其主要目的。我们采用三管齐下的策略来解决小群体公平性量化问题：首先，提出基于"反事实公平性"框架的新估计量，通过跨群体信息共享实现；其次，利用较现有技术更庞大的数据量进行参数估计；最后，创新性地提出一种数据借用法，整合缺乏结果变量与预测值但包含协变量和群体归属信息的"外部数据"。这种对外部数据较低的准入门槛，极大拓展了数据源的可能性。我们以美国中西部某大型医疗系统在COVID-19疫情期间使用的风险预测模型为例，验证了所提估计量的实际应用价值。