While a broad range of techniques have been proposed to tackle distribution shift, the simple baseline of training on an $\textit{undersampled}$ balanced dataset often achieves close to state-of-the-art-accuracy across several popular benchmarks. This is rather surprising, since undersampling algorithms discard excess majority group data. To understand this phenomenon, we ask if learning is fundamentally constrained by a lack of minority group samples. We prove that this is indeed the case in the setting of nonparametric binary classification. Our results show that in the worst case, an algorithm cannot outperform undersampling unless there is a high degree of overlap between the train and test distributions (which is unlikely to be the case in real-world datasets), or if the algorithm leverages additional structure about the distribution shift. In particular, in the case of label shift we show that there is always an undersampling algorithm that is minimax optimal. In the case of group-covariate shift we show that there is an undersampling algorithm that is minimax optimal when the overlap between the group distributions is small. We also perform an experimental case study on a label shift dataset and find that in line with our theory, the test accuracy of robust neural network classifiers is constrained by the number of minority samples.

翻译：尽管已有大量技术被提出来应对分布偏移，但基于$\textit{欠采样}$平衡数据集训练的简单基线方法在多个主流基准测试中往往能达到接近最优的准确率。这一现象颇为令人惊讶，因为欠采样算法会丢弃过量的多数群体数据。为理解这一现象，我们探究学习过程是否从根本上受限于少数群体样本的不足。我们证明，在非参数二分类设定下确实如此。我们的结果表明，在最坏情况下，除非训练分布与测试分布存在高度重叠（这在真实数据集中不太可能发生），或算法利用了关于分布偏移的额外结构信息，否则任何算法的性能都无法超越欠采样。具体而言，在标签偏移情形下，我们证明始终存在一种能达到极小极大最优的欠采样算法。在群体协变量偏移情形下，我们证明当群体分布之间的重叠较小时，存在一种能达到极小极大最优的欠采样算法。我们还对一个标签偏移数据集进行了实验案例研究，结果发现与我们的理论一致：鲁棒神经网络分类器的测试准确率受限于少数样本的数量。