Which parts of a dataset will a given model find difficult? Recent work has shown that SGD-trained models have a bias towards simplicity, leading them to prioritize learning a majority class, or to rely upon harmful spurious correlations. Here, we show that the preference for "easy" runs far deeper: A model may prioritize any class or group of the dataset that it finds simple-at the expense of what it finds complex-as measured by performance difference on the test set. When subsets with different levels of complexity align with demographic groups, we term this difficulty disparity, a phenomenon that occurs even with balanced datasets that lack group/label associations. We show how difficulty disparity is a model-dependent quantity, and is further amplified in commonly-used models as selected by typical average performance scores. We quantify an amplification factor across a range of settings in order to compare disparity of different models on a fixed dataset. Finally, we present two real-world examples of difficulty amplification in action, resulting in worse-than-expected performance disparities between groups even when using a balanced dataset. The existence of such disparities in balanced datasets demonstrates that merely balancing sample sizes of groups is not sufficient to ensure unbiased performance. We hope this work presents a step towards measurable understanding of the role of model bias as it interacts with the structure of data, and call for additional model-dependent mitigation methods to be deployed alongside dataset audits.

翻译：给定模型会认为数据集的哪些部分难以处理？近期研究表明，经SGD训练的模型存在简单性偏好，导致其优先学习多数类，或依赖有害的虚假相关性。本文证明，模型对"简单"样本的偏好远比此更为深远：模型会优先处理数据集中其认为简单的任何类别或子集，并以测试集性能差异为代价，牺牲对复杂样本的学习效果。当不同复杂度子集与人口统计群体一致时，我们将此现象称为难度差异——即便在缺乏群体/标签关联的均衡数据集中也会出现这种状况。我们证明难度差异是模型相关的量，且常被典型平均性能指标所选用的主流模型进一步放大。通过量化多种设定下的放大因子，我们可在固定数据集上比较不同模型的差异程度。最后，我们展示了两个现实世界中的难度放大案例，证实即便使用均衡数据集，群体间的性能差异仍会高于预期。均衡数据集中存在此类差异表明，仅平衡群体的样本量不足以确保无偏性能。我们希望这项研究能推动对模型偏差与数据结构相互作用的可量化理解，并呼吁在数据审计之外，部署更多基于模型的缓解方法。