Algorithms and models are increasingly deployed to inform decisions about people, inevitably affecting their lives. As a consequence, those in charge of developing these models must carefully evaluate their impact on different groups of people and favour group fairness, that is, ensure that groups determined by sensitive demographic attributes, such as race or sex, are not treated unjustly. To achieve this goal, the availability (awareness) of these demographic attributes to those evaluating the impact of these models is fundamental. Unfortunately, collecting and storing these attributes is often in conflict with industry practices and legislation on data minimisation and privacy. For this reason, it can be hard to measure the group fairness of trained models, even from within the companies developing them. In this work, we tackle the problem of measuring group fairness under unawareness of sensitive attributes, by using techniques from quantification, a supervised learning task concerned with directly providing group-level prevalence estimates (rather than individual-level class labels). We show that quantification approaches are particularly suited to tackle the fairness-under-unawareness problem, as they are robust to inevitable distribution shifts while at the same time decoupling the (desirable) objective of measuring group fairness from the (undesirable) side effect of allowing the inference of sensitive attributes of individuals. More in detail, we show that fairness under unawareness can be cast as a quantification problem and solved with proven methods from the quantification literature. We show that these methods outperform previous approaches to measure demographic parity in five experimental protocols, corresponding to important challenges that complicate the estimation of classifier fairness under unawareness.

翻译：算法与模型日益被用于辅助涉及人群的决策，这不可避免地对人们的生活产生影响。因此，负责开发这些模型的人员必须谨慎评估其对不同群体的影响，并促进群体公平性——即确保由种族、性别等敏感人口属性界定的群体不会遭受不公正对待。为实现这一目标，评估模型影响的人员能否获取（即认知）这些人口属性至关重要。然而，收集和存储这些属性往往与行业实践及数据最小化与隐私保护立法相冲突。正因如此，即便在开发这些模型的公司内部，也难以衡量已训练模型的群体公平性。本研究利用量化技术（一种直接提供群体级患病率估计而非个体级类别标签的监督学习任务）来解决在敏感属性不可知条件下衡量群体公平性的问题。我们证明量化方法特别适合处理“不可知条件下的公平性”问题，因其既能应对不可避免的分布偏移，又能将（理想的）衡量群体公平性目标与（非期望的）推断个体敏感属性的副作用相分离。具体而言，我们表明敏感属性不可知条件下的公平性问题可转化为量化问题，并通过量化文献中已有成熟方法解决。我们证明，在对应五个重要挑战的实验协议中，这些方法在衡量人口统计平等性方面优于以往方法，而这五个挑战正是导致分类器公平性估计在不可知条件下复杂化的关键因素。