Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data. However, these models also display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of ``decision shortcuts'' that hinder their generalization capabilities. In this work, we find that the CLIP model possesses a rich set of features, encompassing both \textit{desired invariant causal features} and \textit{undesired decision shortcuts}. Moreover, the underperformance of CLIP on downstream tasks originates from its inability to effectively utilize pre-trained features in accordance with specific task requirements. To address this challenge, we propose a simple yet effective method, Spurious Feature Eraser (SEraser), to alleviate the decision shortcuts by erasing the spurious features. Specifically, we introduce a test-time prompt tuning paradigm that optimizes a learnable prompt, thereby compelling the model to exploit invariant features while disregarding decision shortcuts during the inference phase. The proposed method effectively alleviates excessive dependence on potentially misleading spurious information. We conduct comparative analysis of the proposed method against various approaches which validates the significant superiority.

翻译：视觉语言基础模型因其在海量图文配对数据上的可扩展性，已在众多下游任务中展现出卓越性能。然而，这些模型在下游任务（例如细粒度图像分类）中也表现出明显的局限性，其根源在于“决策捷径”阻碍了模型的泛化能力。本研究发现，CLIP模型拥有丰富的特征集，既包含期望的**不变因果特征**，也包含不期望的**决策捷径**。此外，CLIP在下游任务中表现欠佳的原因在于其无法根据具体任务需求有效利用预训练特征。为应对这一挑战，我们提出了一种简单而有效的方法——伪特征擦除器（SEraser），通过擦除伪特征来缓解决策捷径问题。具体而言，我们引入了一种测试时提示调优范式，通过优化可学习的提示向量，迫使模型在推理阶段利用不变特征而忽略决策捷径。所提方法能有效缓解对潜在误导性伪信息的过度依赖。通过将所提方法与多种基线方案进行对比分析，验证了其显著的优越性。