Existing regulations prohibit model developers from accessing protected attributes (gender, race, etc.), often resulting in fairness assessments on populations without knowing their protected groups. In such scenarios, institutions often adopt a separation between the model developers (who train models with no access to the protected attributes) and a compliance team (who may have access to the entire dataset for auditing purposes). However, the model developers might be allowed to test their models for bias by querying the compliance team for group fairness metrics. In this paper, we first demonstrate that simply querying for fairness metrics, such as statistical parity and equalized odds can leak the protected attributes of individuals to the model developers. We demonstrate that there always exist strategies by which the model developers can identify the protected attribute of a targeted individual in the test dataset from just a single query. In particular, we show that one can reconstruct the protected attributes of all the individuals from O(Nk \log( n /Nk)) queries when Nk<<n using techniques from compressed sensing (n: size of the test dataset, Nk: size of smallest group). Our results pose an interesting debate in algorithmic fairness: should querying for fairness metrics be viewed as a neutral-valued solution to ensure compliance with regulations? Or, does it constitute a violation of regulations and privacy if the number of queries answered is enough for the model developers to identify the protected attributes of specific individuals? To address this supposed violation, we also propose Attribute-Conceal, a novel technique that achieves differential privacy by calibrating noise to the smooth sensitivity of our bias query, outperforming naive techniques such as the Laplace mechanism. We also include experimental results on the Adult dataset and synthetic data.

翻译：现有法规禁止模型开发者获取受保护属性（如性别、种族等），这通常导致在无法知晓受保护群体类别的情况下对人群进行公平性评估。在此类场景中，机构通常采取模型开发者（训练模型时无法接触受保护属性）与合规团队（可能有权访问完整数据集以进行审计）分离的模式。然而，模型开发者可能被允许通过向合规团队查询群体公平性指标来测试其模型的偏差。本文首先证明，仅通过查询公平性指标（如统计均等和均等化几率）就可能导致模型开发者获取个体的受保护属性。我们证明存在某种策略，使得模型开发者仅需单次查询即可识别测试数据集中特定目标的受保护属性。特别地，我们表明当满足Nk<<n条件时（n为测试数据集规模，Nk为最小群体规模），利用压缩感知技术仅需O(Nk log(n/Nk))次查询即可重构所有个体的受保护属性。我们的研究结果引发了算法公平性领域的重要讨论：将公平性指标查询视为确保法规合规的中立解决方案是否合理？或者，当查询次数足以使模型开发者识别特定个体的受保护属性时，这是否构成对法规与隐私的侵犯？针对这种潜在违规，我们提出一种名为Attribute-Conceal的新技术，通过向偏差查询的平滑敏感度添加校准噪声来实现差分隐私，其性能优于拉普拉斯机制等传统方法。我们还提供了在Adult数据集和合成数据上的实验结果。