Machine learning systems are increasingly being used in critical decision making such as healthcare, finance, and criminal justice. Concerns around their fairness have resulted in several bias mitigation techniques that emphasize the need for high-quality data to ensure fairer decisions. However, the role of earlier stages of machine learning pipelines in mitigating model bias has not been explored well. In this paper, we focus on the task of acquiring additional labeled data points for training the downstream machine learning model to rapidly improve its fairness. Since not all data points in a data pool are equally beneficial to the task of fairness, we generate an ordering in which data points should be acquired. We present DataSift, a data acquisition framework based on the idea of data valuation that relies on partitioning and multi-armed bandits to determine the most valuable data points to acquire. Over several iterations, DataSift selects a partition and randomly samples a batch of data points from the selected partition, evaluates the benefit of acquiring the batch on model fairness, and updates the utility of partitions depending on the benefit. To further improve the effectiveness and efficiency of evaluating batches, we leverage influence functions that estimate the effect of acquiring a batch without retraining the model. We empirically evaluate DataSift on several real-world and synthetic datasets and show that the fairness of a machine learning model can be significantly improved even while acquiring a few data points.

翻译：机器学习系统正日益应用于医疗保健、金融和刑事司法等关键决策领域。围绕其公平性的担忧催生了多种偏差缓解技术，这些技术强调需要高质量数据以确保更公平的决策。然而，机器学习流程早期阶段在缓解模型偏差方面的作用尚未得到充分探索。本文重点关注为训练下游机器学习模型获取额外标注数据点以快速提升其公平性的任务。由于数据池中并非所有数据点对公平性任务的贡献均等，我们构建了数据点的优先采集顺序。我们提出DataSift——一种基于数据估值思想的数据采集框架，该框架通过分区技术和多臂老虎机机制确定最具价值的数据点进行采集。经过多次迭代，DataSift会选择一个分区并从选定分区中随机采样一批数据点，评估获取该批次数据对模型公平性的增益，并根据增益结果更新分区的效用值。为提升批次评估的效能与效率，我们采用影响函数来估计获取数据批次的效果而无需重新训练模型。我们在多个真实世界和合成数据集上对DataSift进行了实证评估，结果表明即使仅获取少量数据点，机器学习模型的公平性也能得到显著改善。