Bias can be introduced in diverse ways in machine learning datasets, for example via selection or label bias. Although these bias types in themselves have an influence on important aspects of fair machine learning, their different impact has been understudied. In this work, we empirically analyze the effect of label bias and several subtypes of selection bias on the evaluation of classification models, on their performance, and on the effectiveness of bias mitigation methods. We also introduce a biasing and evaluation framework that allows to model fair worlds and their biased counterparts through the introduction of controlled bias in real-life datasets with low discrimination. Using our framework, we empirically analyze the impact of each bias type independently, while obtaining a more representative evaluation of models and mitigation methods than with the traditional use of a subset of biased data as test set. Our results highlight different factors that influence how impactful bias is on model performance. They also show an absence of trade-off between fairness and accuracy, and between individual and group fairness, when models are evaluated on a test set that does not exhibit unwanted bias. They furthermore indicate that the performance of bias mitigation methods is influenced by the type of bias present in the data. Our findings call for future work to develop more accurate evaluations of prediction models and fairness interventions, but also to better understand other types of bias, more complex scenarios involving the combination of different bias types, and other factors that impact the efficiency of the mitigation methods, such as dataset characteristics.

翻译：在机器学习数据集中，偏差可能以多种方式引入，例如通过选择偏差或标签偏差。尽管这些偏差类型本身对机器学习公平性的重要方面具有影响，但它们的差异影响尚未得到充分研究。在本工作中，我们通过实证分析标签偏差及多种选择偏差子类型对分类模型评估、模型性能以及偏差缓解方法有效性的影响。我们还引入了一个偏差构建与评估框架，该框架能够通过在低歧视度的现实数据集中引入受控偏差，来模拟公平世界及其对应的偏差版本。利用该框架，我们独立地实证分析了每种偏差类型的影响，同时获得了比传统使用偏差数据子集作为测试集更具代表性的模型与缓解方法评估结果。我们的研究结果揭示了影响偏差对模型性能作用程度的不同因素。结果表明，当在未呈现非期望偏差的测试集上评估模型时，公平性与准确性之间以及个体公平性与群体公平性之间不存在权衡关系。研究进一步表明，偏差缓解方法的性能受到数据中存在的偏差类型的影响。我们的发现呼吁未来研究致力于开发更精准的预测模型与公平性干预措施评估方法，同时更深入地理解其他偏差类型、涉及多种偏差类型组合的复杂场景，以及影响缓解方法效率的其他因素（如数据集特征）。