We study the problem of auditing the fairness of a given classifier under partial feedback, where true labels are available only for positively classified individuals, (e.g., loan repayment outcomes are observed only for approved applicants). We introduce a novel cost model for acquiring additional labeled data, designed to more accurately reflect real-world costs such as credit assessment, loan processing, and potential defaults. Our goal is to find optimal fairness audit algorithms that are more cost-effective than random exploration and natural baselines. In our work, we consider two audit settings: a black-box model with no assumptions on the data distribution, and a mixture model, where features and true labels follow a mixture of exponential family distributions. In the black-box setting, we propose a near-optimal auditing algorithm under mild assumptions and show that a natural baseline can be strictly suboptimal. In the mixture model setting, we design a novel algorithm that achieves significantly lower audit cost than the black-box case. Our approach leverages prior work on learning from truncated samples and maximum-a-posteriori oracles, and extends known results on spherical Gaussian mixtures to handle exponential family mixtures, which may be of independent interest. Moreover, our algorithms apply to popular fairness metrics including demographic parity, equal opportunity, and equalized odds. Empirically, we demonstrate strong performance of our algorithms on real-world fair classification datasets like Adult Income and Law School, consistently outperforming natural baselines by around 50% in terms of audit cost.

翻译：本研究探讨在部分反馈条件下审计给定分类器公平性的问题，其中真实标签仅对正向分类的个体可用（例如，仅对获批申请人可观测贷款偿还结果）。我们引入了一种获取额外标注数据的新型成本模型，旨在更准确地反映现实世界成本，如信用评估、贷款处理及潜在违约。我们的目标是寻找比随机探索和自然基线方法更具成本效益的最优公平性审计算法。工作中我们考虑两种审计场景：不对数据分布做任何假设的黑盒模型，以及特征与真实标签遵循指数族分布混合的混合模型。在黑盒场景下，我们在温和假设下提出了一种近似最优的审计算法，并证明自然基线方法可能严格次优。在混合模型场景中，我们设计了一种新算法，其审计成本显著低于黑盒情况。我们的方法借鉴了从截断样本学习及最大后验预言机的现有研究，并将球形高斯混合的已知结果扩展至可处理指数族混合，这可能具有独立研究价值。此外，我们的算法适用于包括人口统计均等、机会均等和几率均等在内的主流公平性指标。实证方面，我们在成人收入与法学院等现实世界公平分类数据集上验证了算法的优异性能，其审计成本持续优于自然基线方法约50%。