Pre-trained vision-language models (VLMs) have achieved promising success in many fields, especially with prompt learning paradigm. In this work, we propose GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better explore the compositional zero-shot learning (CZSL) ability of VLMs within the prompt-based learning framework. The soft prompt in GIPCOL is structured and consists of the prefix learnable vectors, attribute label and object label. In addition, the attribute and object labels in the soft prompt are designated as nodes in a compositional graph. The compositional graph is constructed based on the compositional structure of the objects and attributes extracted from the training data and consequently feeds the updated concept representation into the soft prompt to capture this compositional structure for a better prompting for CZSL. With the new prompting strategy, GIPCOL achieves state-of-the-art AUC results on all three CZSL benchmarks, including MIT-States, UT-Zappos, and C-GQA datasets in both closed and open settings compared to previous non-CLIP as well as CLIP-based methods. We analyze when and why GIPCOL operates well given the CLIP backbone and its training data limitations, and our findings shed light on designing more effective prompts for CZSL

翻译：预训练视觉语言模型（VLM）在许多领域已取得显著成功，尤其是在提示学习范式下。本文提出GIPCOL（基于图注入的软提示组合学习），旨在基于提示学习框架中更好地探索VLM的组合式零样本学习（CZSL）能力。GIPCOL中的软提示是结构化的，由前缀可学习向量、属性标签和对象标签组成。此外，软提示中的属性标签和对象标签被指定为组合图中的节点。该组合图基于从训练数据中提取的对象与属性的组合结构构建，进而将更新后的概念表示注入软提示，以捕获这种组合结构，从而为CZSL提供更优的提示策略。凭借新的提示策略，GIPCOL在所有三个CZSL基准数据集（包括MIT-States、UT-Zappos和C-GQA数据集）的封闭与开放设置下，相较于先前基于非CLIP及CLIP的方法，均取得了最先进的AUC结果。我们分析了在CLIP骨干网络及其训练数据限制条件下GIPCOL何时及为何能良好运行，这些发现为设计更有效的CZSL提示提供了启示。