While methods for measuring and correcting differential performance in risk prediction models have proliferated in recent years, most existing techniques can only be used to assess fairness across relatively large subgroups. The purpose of algorithmic fairness efforts is often to redress discrimination against groups that are both marginalized and small, so this sample size limitation often prevents existing techniques from accomplishing their main aim. We take a three-pronged approach to address the problem of quantifying fairness with small subgroups. First, we propose new estimands built on the "counterfactual fairness" framework that leverage information across groups. Second, we estimate these quantities using a larger volume of data than existing techniques. Finally, we propose a novel data borrowing approach to incorporate "external data" that lacks outcomes and predictions but contains covariate and group membership information. This less stringent requirement on the external data allows for more possibilities for external data sources. We demonstrate practical application of our estimators to a risk prediction model used by a major Midwestern health system during the COVID-19 pandemic.

翻译：近年来，衡量和校正风险预测模型中差异化性能的方法不断涌现，但现有技术大多仅适用于评估相对较大子群的公平性。算法公平性工作的目标通常是纠正对边缘化且规模较小的群体的歧视，因此这种样本量限制往往阻碍现有技术实现其主要目的。我们采用三管齐下的方法来解决小子群公平性量化问题：首先，基于"反事实公平性"框架提出利用跨群体信息的新估计量；其次，通过比现有技术更大量的数据来估计这些量；最后，提出一种新颖的数据借用法，以整合缺乏结果和预测但包含协变量与群体成员信息的"外部数据"。这种对外部数据的宽松要求拓展了外部数据源的可能性。我们将所提估计量应用于美国中西部某大型卫生系统在新冠疫情期使用的风险预测模型，展示了其实践应用价值。