Neural image classifiers can often learn to make predictions by overly relying on non-predictive features that are spuriously correlated with the class labels in the training data. This leads to poor performance in real-world atypical scenarios where such features are absent. Supplementing the training dataset with images without such spurious features can aid robust learning against spurious correlations via better generalization. This paper presents ASPIRE (Language-guided data Augmentation for SPurIous correlation REmoval), a simple yet effective solution for expanding the training dataset with synthetic images without spurious features. ASPIRE, guided by language, generates these images without requiring any form of additional supervision or existing examples. Precisely, we employ LLMs to first extract foreground and background features from textual descriptions of an image, followed by advanced language-guided image editing to discover the features that are spuriously correlated with the class label. Finally, we personalize a text-to-image generation model to generate diverse in-domain images without spurious features. We demonstrate the effectiveness of ASPIRE on 4 datasets, including the very challenging Hard ImageNet dataset, and 9 baselines and show that ASPIRE improves the classification accuracy of prior methods by 1% - 38%. Code soon at: https://github.com/Sreyan88/ASPIRE.

翻译：神经图像分类器常因过度依赖训练数据中与类别标签存在虚假相关性的非预测特征而学习到错误的预测模式。当真实场景中此类特征缺失时，模型表现显著下降。通过补充不含虚假特征的图像扩展训练数据集，可有效提升模型对虚假相关性的鲁棒泛化能力。本文提出ASPIRE（基于语言引导的虚假相关性去除数据增强方法），这是一种简洁高效的解决方案，通过生成不含虚假特征的合成图像扩展训练数据集。该方法仅需语言引导，无需额外监督信号或现成示例即可生成图像。具体而言，我们首先利用大语言模型（LLM）从图像的文本描述中提取前景与背景特征，随后通过先进的语言引导图像编辑技术，发现与类别标签存在虚假相关性的特征。最后，我们个性化定制文本到图像生成模型，生成领域内多样化且不含虚假特征的图像。我们在4个数据集（含极具挑战性的Hard ImageNet数据集）及9个基线模型上验证了ASPIRE的有效性，结果表明该方法可将现有方法的分类准确率提升1%-38%。代码即将开源：https://github.com/Sreyan88/ASPIRE。