CLIP (Contrastive Language-Image Pretraining) is well-developed for open-vocabulary zero-shot image-level recognition, while its applications in pixel-level tasks are less investigated, where most efforts directly adopt CLIP features without deliberative adaptations. In this work, we first demonstrate the necessity of image-pixel CLIP feature adaption, then provide Multi-View Prompt learning (MVP-SEG) as an effective solution to achieve image-pixel adaptation and to solve open-vocabulary semantic segmentation. Concretely, MVP-SEG deliberately learns multiple prompts trained by our Orthogonal Constraint Loss (OCLoss), by which each prompt is supervised to exploit CLIP feature on different object parts, and collaborative segmentation masks generated by all prompts promote better segmentation. Moreover, MVP-SEG introduces Global Prompt Refining (GPR) to further eliminate class-wise segmentation noise. Experiments show that the multi-view prompts learned from seen categories have strong generalization to unseen categories, and MVP-SEG+ which combines the knowledge transfer stage significantly outperforms previous methods on several benchmarks. Moreover, qualitative results justify that MVP-SEG does lead to better focus on different local parts.

翻译：CLIP（对比语言-图像预训练）在开放词汇的零样本图像级识别任务中发展成熟，但其在像素级任务中的应用研究较少——现有工作大多直接采用CLIP特征而未进行精细调整。本文首先论证了图像像素级CLIP特征适配的必要性，随后提出多视角提示学习（MVP-SEG）作为实现图像像素适配并解决开放词汇语义分割的有效方案。具体而言，MVP-SEG通过我们提出的正交约束损失（OCLoss）训练多个提示，每个提示被监督学习以挖掘CLIP在不同对象部位的特征，所有提示生成的协作分割掩码可提升分割效果。此外，MVP-SEG引入全局提示精炼（GPR）进一步消除类别级分割噪声。实验表明，从可见类别学习的多视角提示对未见类别具有强泛化能力，而集成知识迁移阶段的MVP-SEG+在多个基准上显著优于现有方法。定性结果进一步证实MVP-SEG能更好地聚焦不同局部区域。