Evaluating machine learning (ML) model bias is key to building trustworthy and robust ML systems. Counterfactual Fairness (CF) audits allow the measurement of bias of ML models with a causal framework, yet their conclusions rely on a single causal graph that is rarely known with certainty in real-world scenarios. We propose CF with Graph Uncertainty (CF-GU), a bias evaluation procedure that incorporates the uncertainty of specifying a causal graph into CF. CF-GU (i) bootstraps a Causal Discovery algorithm under domain knowledge constraints to produce a bag of plausible Directed Acyclic Graphs (DAGs), (ii) quantifies graph uncertainty with the normalized Shannon entropy, and (iii) provides confidence bounds on CF metrics. Experiments on synthetic data show how contrasting domain knowledge assumptions support or refute audits of CF, while experiments on real-world data (COMPAS and Adult datasets) pinpoint well-known biases with high confidence, even when supplied with minimal domain knowledge constraints.

翻译：评估机器学习（ML）模型偏差是构建可信赖且鲁棒的ML系统的关键。反事实公平性（CF）审计允许通过因果框架衡量ML模型的偏差，但其结论依赖于单一的因果图，而在现实场景中因果图很少能被确定性地获知。我们提出基于图不确定性的反事实公平性（CF-GU），这是一种将因果图指定的不确定性纳入CF的偏差评估方法。CF-GU（i）在领域知识约束下对因果发现算法进行自助采样，生成一组合理的无环有向图（DAGs）；（ii）使用归一化香农熵量化图不确定性；（iii）为CF指标提供置信区间。在合成数据上的实验展示了不同的领域知识假设如何支持或反驳CF审计结论，而在真实世界数据（COMPAS和Adult数据集）上的实验则能以高置信度识别已知偏差，即使在仅提供最小领域知识约束的情况下亦如此。