The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example, a classifier may misclassify dog breeds based on the background of dog images. This happens when the backgrounds are correlated with other breeds in the training data, leading to misclassifications during test time. Previous SC benchmark datasets suffer from varying issues, e.g., over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. In this paper, we present \benchmark-\{O2O, M2M\}-\{Easy, Medium, Hard\}, an image classification benchmark suite containing spurious correlations between classes and backgrounds. To create this dataset, we employ a text-to-image model to generate photo-realistic images and an image captioning model to filter out unsuitable ones. The resulting dataset is of high quality and contains approximately 152k images. Our experimental results demonstrate that state-of-the-art group robustness methods struggle with \benchmark, most notably on the Hard-splits with none of them getting over $70\%$ accuracy on the hardest split using a ResNet50 pretrained on ImageNet. By examining model misclassifications, we detect reliances on spurious backgrounds, demonstrating that our dataset provides a significant challenge.

翻译：虚假相关性（Spurious Correlations, SC）问题源于分类器依赖训练数据中与标签偶然相关的非预测性特征。例如，当狗图像背景与训练数据中其他品种存在相关性时，分类器可能基于背景错误识别犬种。现有SC基准数据集存在不同缺陷，如过度饱和或仅包含一对一（O2O）SC而缺乏虚假属性与类别组间产生的多对多（M2M）SC。本文提出\benchmark-\{O2O, M2M\}-\{易, 中, 难\}图像分类基准套件，聚焦类别与背景间的虚假相关性。我们通过文本到图像模型生成逼真图像，并采用图像描述模型筛选不合格样本，构建了含约15.2万张高质量图像的数据集。实验表明，现有最先进组鲁棒性方法在处理\benchmark时均面临挑战——尤其在困难划分中，使用ImageNet预训练ResNet50的模型在最具挑战的划分上准确率均未超过70%。通过分析模型误分类案例，我们检测到模型对虚假背景特征的依赖，证实本数据集构成了显著挑战。