Auditing social-media algorithms has become a focus of public-interest research and policymaking to ensure their fairness across demographic groups such as race, age, and gender in consequential domains such as the presentation of employment opportunities. However, such demographic attributes are often unavailable to auditors and platforms. When demographics data is unavailable, auditors commonly infer them from other available information. In this work, we study the effects of inference error on auditing for bias in one prominent application: black-box audit of ad delivery using paired ads. We show that inference error, if not accounted for, causes auditing to falsely miss skew that exists. We then propose a way to mitigate the inference error when evaluating skew in ad delivery algorithms. Our method works by adjusting for expected error due to demographic inference, and it makes skew detection more sensitive when attributes must be inferred. Because inference is increasingly used for auditing, our results provide an important addition to the auditing toolbox to promote correct audits of ad delivery algorithms for bias. While the impact of attribute inference on accuracy has been studied in other domains, our work is the first to consider it for black-box evaluation of ad delivery bias, when only aggregate data is available to the auditor.

翻译：社交媒体算法的审计已成为公共利益研究和政策制定的焦点，以确保其在就业机会呈现等重要领域中，在不同人口群体（如种族、年龄和性别）之间保持公平性。然而，审计者和平台通常无法获取这些人口属性数据。当人口统计数据不可用时，审计者通常从其他可用信息中推断这些属性。在本研究中，我们探讨了推断误差对一项重要应用中的偏见审计的影响：使用配对广告进行广告投放的黑盒审计。我们证明，如果不考虑推断误差，审计将错误地忽略实际存在的偏差。随后，我们提出了一种在评估广告投放算法偏差时减轻推断误差的方法。该方法通过调整人口属性推断带来的预期误差，在必须推断属性时提高偏差检测的敏感性。由于推断在审计中的应用日益广泛，我们的研究成果为审计工具箱提供了重要补充，以促进对广告投放算法偏见的正确审计。尽管属性推断对准确性的影响已在其他领域得到研究，但我们的工作首次在审计者仅能获取聚合数据的情况下，将其应用于广告投放偏差的黑盒评估。