Online platforms mediate access to opportunity: relevance-based rankings create and constrain options by allocating exposure to job openings and job candidates in hiring platforms, or sellers in a marketplace. In order to do so responsibly, these socially consequential systems employ various fairness measures and interventions, many of which seek to allocate exposure based on worthiness. Because these constructs are typically not directly observable, platforms must instead resort to using proxy scores such as relevance and infer them from behavioral signals such as searcher clicks. Yet, it remains an open question whether relevance fulfills its role as such a worthiness score in high-stakes fair rankings. In this paper, we combine perspectives and tools from the social sciences, information retrieval, and fairness in machine learning to derive a set of desired criteria that relevance scores should satisfy in order to meaningfully guide fairness interventions. We then empirically show that not all of these criteria are met in a case study of relevance inferred from biased user click data. We assess the impact of these violations on the estimated system fairness and analyze whether existing fairness interventions may mitigate the identified issues. Our analyses and results surface the pressing need for new approaches to relevance collection and generation that are suitable for use in fair ranking.

翻译：在线平台中介着获取机会的渠道：基于相关性的排序通过分配对招聘平台中职位空缺和求职者、或市场中卖家的曝光度，来创造和限制选择。为了负责任地行事，这些具有社会影响的系统采用了各种公平性指标和干预措施，其中许多旨在根据应得性分配曝光度。由于这些构建指标通常无法直接观测，平台必须转而使用相关性等代理分数，并从用户点击等行为信号中推断出来。然而，相关性是否在高风险的公平排序中履行了其作为应得性分数的角色，仍是一个悬而未决的问题。在本文中，我们结合社会科学、信息检索和机器学习公平性领域的视角与工具，推导出一组相关性分数应满足的期望标准，以便有意义地指导公平性干预措施。随后，我们通过一项从有偏用户点击数据中推断相关性的案例研究，实证展示了并非所有标准都得到满足。我们评估了这些违反行为对系统公平性估算的影响，并分析了现有公平性干预措施能否缓解已识别的问题。我们的分析和结果揭示了迫切需要开发适用于公平排序的新型相关性收集与生成方法。