Evaluating fairness can be challenging in practice because the sensitive attributes of data are often inaccessible due to privacy constraints. The go-to approach that the industry frequently adopts is using off-the-shelf proxy models to predict the missing sensitive attributes, e.g. Meta [Alao et al., 2021] and Twitter [Belli et al., 2022]. Despite its popularity, there are three important questions unanswered: (1) Is directly using proxies efficacious in measuring fairness? (2) If not, is it possible to accurately evaluate fairness using proxies only? (3) Given the ethical controversy over inferring user private information, is it possible to only use weak (i.e. inaccurate) proxies in order to protect privacy? Our theoretical analyses show that directly using proxy models can give a false sense of (un)fairness. Second, we develop an algorithm that is able to measure fairness (provably) accurately with only three properly identified proxies. Third, we show that our algorithm allows the use of only weak proxies (e.g. with only 68.85%accuracy on COMPAS), adding an extra layer of protection on user privacy. Experiments validate our theoretical analyses and show our algorithm can effectively measure and mitigate bias. Our results imply a set of practical guidelines for practitioners on how to use proxies properly. Code is available at github.com/UCSC-REAL/fair-eval.

翻译：在实际评估公平性时，由于隐私限制，数据的敏感属性往往难以获取，这带来了挑战。业界常用方法是使用现成的代理模型预测缺失的敏感属性，例如Meta [Alao et al., 2021]和Twitter [Belli et al., 2022]的做法。尽管该方法广泛使用，仍有三个重要问题尚未解答：(1) 直接使用代理模型能否有效衡量公平性？(2) 如果不能，仅使用代理模型是否可能准确评估公平性？(3) 鉴于推断用户隐私信息引发的伦理争议，是否可能仅使用弱（即不准确）代理来保护隐私？我们的理论分析表明，直接使用代理模型可能产生对（不）公平的虚假感知。其次，我们开发了一种算法，仅需三个适当识别的代理即可（可证明地）准确测量公平性。第三，我们证明该算法允许仅使用弱代理（例如在COMPAS上仅68.85%的准确率），从而为用户隐私增加额外保护层。实验验证了我们的理论分析，并表明该算法能有效测量和缓解偏差。研究结果为从业人员合理使用代理提供了一套实用指南。代码已开源：github.com/UCSC-REAL/fair-eval。