AI fairness measurements, including tests for equal treatment, often take the form of disaggregated evaluations of AI systems. Such measurements are an important part of Responsible AI operations. These measurements compare system performance across demographic groups or sub-populations and typically require member-level demographic signals such as gender, race, ethnicity, and location. However, sensitive member-level demographic attributes like race and ethnicity can be challenging to obtain and use due to platform choices, legal constraints, and cultural norms. In this paper, we focus on the task of enabling AI fairness measurements on race/ethnicity for \emph{U.S. LinkedIn members} in a privacy-preserving manner. We present the Privacy-Preserving Probabilistic Race/Ethnicity Estimation (PPRE) method for performing this task. PPRE combines the Bayesian Improved Surname Geocoding (BISG) model, a sparse LinkedIn survey sample of self-reported demographics, and privacy-enhancing technologies like secure two-party computation and differential privacy to enable meaningful fairness measurements while preserving member privacy. We provide details of the PPRE method and its privacy guarantees. We then illustrate sample measurement operations. We conclude with a review of open research and engineering challenges for expanding our privacy-preserving fairness measurement capabilities.

翻译：人工智能公平性测量（包括平等对待测试）通常表现为对AI系统的分群体评估。此类测量是负责任AI运营的重要组成部分。这些测量通过比较不同人口统计群体或亚群体间的系统性能，通常需要成员级别的性别、种族、族裔和地理位置等人口统计信号。然而，由于平台选择、法律限制和文化规范等因素，种族和族裔等敏感成员级人口属性往往难以获取和使用。本文聚焦于以隐私保护方式实现对\emph{美国LinkedIn会员}进行种族/族裔公平性测量的任务。我们提出了用于此任务的隐私保护概率型种族/族裔估计（PPRE）方法。PPRE融合了贝叶斯改进姓氏地理编码（BISG）模型、LinkedIn稀疏调查样本中的自报人口统计数据，以及安全两方计算与差分隐私等隐私增强技术，在保护会员隐私的同时实现有意义的公平性测量。我们详细阐述了PPRE方法及其隐私保障机制，并通过示例演示测量操作流程。最后，我们对拓展隐私保护型公平性测量能力所面临的开放研究与工程挑战进行了评述。