Unfair predictions of machine learning (ML) models impede their broad acceptance in real-world settings. Tackling this arduous challenge first necessitates defining what it means for an ML model to be fair. This has been addressed by the ML community with various measures of fairness that depend on the prediction outcomes of the ML models, either at the group level or the individual level. These fairness measures are limited in that they utilize point predictions, neglecting their variances, or uncertainties, making them susceptible to noise, missingness and shifts in data. In this paper, we first show that an ML model may appear to be fair with existing point-based fairness measures but biased against a demographic group in terms of prediction uncertainties. Then, we introduce new fairness measures based on different types of uncertainties, namely, aleatoric uncertainty and epistemic uncertainty. We demonstrate on many datasets that (i) our uncertainty-based measures are complementary to existing measures of fairness, and (ii) they provide more insights about the underlying issues leading to bias.

翻译：机器学习模型的不公平预测阻碍了其在现实场景中的广泛接受。解决这一艰巨挑战首先需要明确机器学习模型公平性的定义。机器学习社区已通过多种依赖于模型预测结果的公平性度量来应对该问题，这些度量既涵盖群体层面也包含个体层面。现有公平性度量存在局限性，因为它们仅利用点预测值而忽视其方差（即不确定性），这使得它们易受数据噪声、缺失和分布偏移的影响。本文首先证明，基于现有基于点预测的公平性度量，机器学习模型看似公平，但可能在预测不确定性方面对某人口群体存在偏见。随后，我们引入基于不同不确定性类型（即偶然不确定性和认知不确定性）的新型公平性度量。我们在多个数据集上证明：（i）基于不确定性的度量是对现有公平性度量的补充；（ii）它们能提供关于导致偏差的潜在问题的更多见解。