We introduce compositional soft prompting (CSP), a parameter-efficient learning technique to improve the zero-shot compositionality of large-scale pretrained vision-language models (VLMs) like CLIP. We develop CSP for compositional zero-shot learning, the task of predicting unseen attribute-object compositions (e.g., old cat and young tiger). VLMs have a flexible text encoder that can represent arbitrary classes as natural language prompts but they often underperform task-specific architectures on the compositional zero-shot benchmark datasets. CSP treats the attributes and objects that define classes as learnable tokens of vocabulary. During training, the vocabulary is tuned to recognize classes that compose tokens in multiple ways (e.g., old cat and white cat). At test time, we recompose the learned attribute-object vocabulary in new combinations to recognize novel classes. We show that CSP outperforms the CLIP on benchmark datasets by an average of 10.9 percentage points on AUC. CSP also outperforms CoOp, a soft prompting method that fine-tunes the prefix context tokens, by an average of 5.8 percentage points on AUC. We perform additional experiments to show that CSP improves generalization to higher-order attribute-attribute-object compositions (e.g., old white cat) and combinations of pretrained attributes and fine-tuned objects. The code is available at https://github.com/BatsResearch/csp.

翻译：我们提出组合软提示（CSP）技术，这是一种参数高效的学习方法，旨在提升大规模预训练视觉语言模型（如CLIP）在零样本组合任务中的性能。CSP专为组合式零样本学习设计——即预测未见过的属性-物体组合（例如"老猫"和"幼虎"）。视觉语言模型具备灵活的文本编码器，可通过自然语言提示表征任意类别，但在组合式零样本基准数据集上，其表现通常逊于任务专用架构。CSP将定义类别的属性与物体视为可学习的词汇标记。在训练阶段，该词汇表通过调整以识别多方式组合的类别（如"老猫"和"白猫"）；测试时，我们将已学习的属性-物体词汇重新组合为新组合，以识别未见类别。实验表明，在基准数据集上，CSP在AUC指标上平均超越CLIP 10.9个百分点，并平均超越软提示方法CoOp（该方法仅微调前缀上下文标记）5.8个百分点。进一步实验证明，CSP能提升对高阶属性-属性-物体组合（如"老白猫"）以及预训练属性与微调物体混合组合的泛化能力。代码已开源至 https://github.com/BatsResearch/csp。