The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example, a classifier may misclassify dog breeds based on the background of dog images. This happens when the backgrounds are correlated with other breeds in the training data, leading to misclassifications during test time. Previous SC benchmark datasets suffer from varying issues, e.g., over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. In this paper, we present Spawrious-{O2O, M2M}-{Easy, Medium, Hard}, an image classification benchmark suite containing spurious correlations among different dog breeds and background locations. To create this dataset, we employ a text-to-image model to generate photo-realistic images, and an image captioning model to filter out unsuitable ones. The resulting dataset is of high quality, containing approximately 152,000 images. Our experimental results demonstrate that state-of-the-art group robustness methods struggle with Spawrious, most notably on the Hard-splits with $<60\%$ accuracy. By examining model misclassifications, we detect reliances on spurious backgrounds, demonstrating that our dataset provides a significant challenge to drive future research.

翻译：虚假相关问题源于分类器依赖训练数据中与标签偶然相关的非预测性特征。例如，分类器可能基于狗图像的背景错误识别犬种。当训练数据中背景与其他犬种存在相关性时，测试阶段便会出现误分类。现有虚假相关性基准数据集存在不同问题，例如过度饱和或仅包含一对一（O2O）的虚假相关性，缺乏虚假属性与类别组间出现的多对多（M2M）相关性。本文提出Spawrious-{O2O, M2M}-{Easy, Medium, Hard}图像分类基准套件，聚焦不同犬种与背景位置间的虚假相关性。我们采用文本到图像模型生成照片级真实图像，并利用图像描述模型过滤不合格样本，构建了包含约152,000张高质量图像的数据集。实验结果表明，最先进的群体鲁棒性方法在Spawrious上表现欠佳，尤其在Hard划分中准确率低于<60%。通过分析模型误分类情况，我们检测到其对虚假背景的依赖，证明该数据集为驱动未来研究提供了重要挑战。