Ranking is a ubiquitous method for focusing the attention of human evaluators on a manageable subset of options. Its use as part of human decision-making processes ranges from surfacing potentially relevant products on an e-commerce site to prioritizing college applications for human review. While ranking can make human evaluation more effective by focusing attention on the most promising options, we argue that it can introduce unfairness if the uncertainty of the underlying relevance model differs between groups of options. Unfortunately, such disparity in uncertainty appears widespread, often to the detriment of minority groups for which relevance estimates can have higher uncertainty due to a lack of data or appropriate features. To address this fairness issue, we propose Equal-Opportunity Ranking (EOR) as a new fairness criterion for ranking and show that it corresponds to a group-wise fair lottery among the relevant options even in the presence of disparate uncertainty. EOR optimizes for an even cost burden on all groups, unlike the conventional Probability Ranking Principle, and is fundamentally different from existing notions of fairness in rankings, such as demographic parity and proportional Rooney rule constraints that are motivated by proportional representation relative to group size. To make EOR ranking practical, we present an efficient algorithm for computing it in time $O(n \log(n))$ and prove its close approximation guarantee to the globally optimal solution. In a comprehensive empirical evaluation on synthetic data, a US Census dataset, and a real-world audit of Amazon search queries, we find that the algorithm reliably guarantees EOR fairness while providing effective rankings.

翻译：排序是一种普遍采用的方法，用于将人类评估者的注意力集中在可管理的选项子集上。作为人类决策过程的一部分，其应用范围广泛，从电子商务网站上展示潜在相关产品，到优先处理需要人工审核的大学申请。虽然排序可以通过将注意力集中在最有希望的选项上，使人类评估更有效，但我们认为，如果底层相关性模型在不同选项组之间存在不确定性差异，排序就可能引入不公平。不幸的是，这种不确定性的差异普遍存在，并且往往对少数群体不利，因为由于缺乏数据或适当特征，这些群体的相关性估计可能具有更高的不确定性。为了解决这一公平性问题，我们提出了机会均等排序作为一种新的排序公平性标准，并证明即使在存在差异不确定性的情况下，它也对应于相关选项之间的组间公平抽签。与传统的概率排序原则不同，EOR 旨在优化所有群体的成本负担均等，并且与现有的排序公平性概念（例如基于相对于群体规模的比例代表性的人口统计均等和比例鲁尼规则约束）有根本区别。为了使 EOR 排序实用化，我们提出了一种在 $O(n \log(n))$ 时间内计算它的高效算法，并证明了其对全局最优解的紧密逼近保证。在合成数据、美国人口普查数据集以及对亚马逊搜索查询的实际审计进行的全面实证评估中，我们发现该算法在提供有效排序的同时，可靠地保证了 EOR 公平性。