A commonly observed pattern in machine learning models is an underprediction of the target feature, with the model's predicted target rate for members of a given category typically being lower than the actual target rate for members of that category in the training set. This underprediction is usually larger for members of minority groups; while income level is underpredicted for both men and women in the 'adult' dataset, for example, the degree of underprediction is significantly higher for women (a minority in that dataset). We propose that this pattern of underprediction for minorities arises as a predictable consequence of statistical inference on small samples. When presented with a new individual for classification, an ML model performs inference not on the entire training set, but on a subset that is in some way similar to the new individual, with sizes of these subsets typically following a power law distribution so that most are small (and with these subsets being necessarily smaller for the minority group). We show that such inference on small samples is subject to systematic and directional statistical bias, and that this bias produces the observed patterns of underprediction seen in ML models. Analysing a standard sklearn decision tree model's predictions on a set of over 70 subsets of the 'adult' and COMPAS datasets, we found that a bias prediction measure based on small-sample inference had a significant positive correlations (0.56 and 0.85) with the observed underprediction rate for these subsets.

翻译：机器学习模型中的一个常见模式是对目标特征的预测偏低，即模型对给定类别成员的预测目标率通常低于训练集中该类别的实际目标率。这种预测偏低现象在少数群体成员中通常更为显著；例如，在“成人”数据集中，虽然男性和女性的收入水平均被低估，但女性的低估程度（该数据集中的少数群体）明显更高。我们提出，这种对少数群体的预测偏低模式是统计推断在小样本上的可预测结果。当对新的个体进行分类时，机器学习模型并非基于整个训练集进行推断，而是基于与该个体在某种程度上相似的一个子集，这些子集的大小通常遵循幂律分布，因此大多数子集较小（且少数群体的子集必然更小）。我们证明了此类小样本推断存在系统性的方向性统计偏差，并且这种偏差导致了机器学习模型中观察到的预测偏低模式。通过分析标准sklearn决策树模型在“成人”和COMPAS数据集上超过70个子集的预测结果，我们发现基于小样本推断的偏差预测度量与这些子集上观察到的预测偏低率存在显著正相关（分别为0.56和0.85）。