Biased data can lead to unfair machine learning models, highlighting the importance of embedding fairness at the beginning of data analysis, particularly during dataset curation and labeling. In response, we propose Falcon, a scalable fair active learning framework. Falcon adopts a data-centric approach that improves machine learning model fairness via strategic sample selection. Given a user-specified group fairness measure, Falcon identifies samples from "target groups" (e.g., (attribute=female, label=positive)) that are the most informative for improving fairness. However, a challenge arises since these target groups are defined using ground truth labels that are not available during sample selection. To handle this, we propose a novel trial-and-error method, where we postpone using a sample if the predicted label is different from the expected one and falls outside the target group. We also observe the trade-off that selecting more informative samples results in higher likelihood of postponing due to undesired label prediction, and the optimal balance varies per dataset. We capture the trade-off between informativeness and postpone rate as policies and propose to automatically select the best policy using adversarial multi-armed bandit methods, given their computational efficiency and theoretical guarantees. Experiments show that Falcon significantly outperforms existing fair active learning approaches in terms of fairness and accuracy and is more efficient. In particular, only Falcon supports a proper trade-off between accuracy and fairness where its maximum fairness score is 1.8-4.5x higher than the second-best results.

翻译：有偏数据可能导致不公平的机器学习模型，这凸显了在数据分析初期——特别是数据集整理与标注阶段——融入公平性的重要性。为此，我们提出Falcon，一个可扩展的公平主动学习框架。Falcon采用数据驱动方法，通过策略性样本选择提升机器学习模型的公平性。针对用户指定的群体公平性指标，Falcon从"目标群体"（如（性别=女性，标签=正类））中识别对改善公平性最具信息量的样本。然而，由于这些目标群体依赖样本选择阶段不可获取的真实标签进行定义，这带来了挑战。为应对此问题，我们提出一种新颖的试错方法：当样本的预测标签与期望标签不同且不属于目标群体时，延迟使用该样本。我们还观察到一种权衡：选择更具信息量的样本会导致因标签预测不合预期而更可能被延迟，且最优平衡点因数据集而异。我们将该权衡视为策略（权衡信息性与延迟率之间），并基于计算效率与理论保障，提出利用对抗式多臂老虎机方法自动选择最优策略。实验表明，Falcon在公平性与准确性上显著优于现有公平主动学习方法，且效率更高。特别地，仅Falcon能实现准确性与公平性的恰当权衡，其最大公平性得分比次优结果高出1.8-4.5倍。