The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both worlds and propose a zero-shot open-vocabulary semantic segmentation method, which does not require any annotations. We propose to locally improve dense MaskCLIP features, computed with a simple modification of CLIP's last pooling layer, by integrating localization priors extracted from self-supervised features. By doing so, we greatly improve the performance of MaskCLIP and produce smooth outputs. Moreover, we show that the used self-supervised feature properties can directly be learnt from CLIP features therefore allowing us to obtain the best results with a single pass through CLIP model. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference, no extra supervision nor extra memory and reaches state-of-the-art results on challenging and fine-grained benchmarks such as COCO, Pascal Context, Cityscapes and ADE20k. The code to reproduce our results is available at https://github.com/wysoczanska/clip_dinoiser.

翻译：摘要：广受欢迎的CLIP模型凭借其与任意文本提示的无缝交互，展现出令人瞩目的零样本能力。然而，其缺乏空间感知能力，使其在无需额外微调步骤的情况下不适用于密集计算机视觉任务（如语义分割），而这类微调通常需要使用标注数据并可能抑制其原有的开放词汇特性。与此同时，自监督表示方法无需人工标注或显式监督即可展现良好的定位能力。在本工作中，我们取二者之长，提出了一种无需任何标注的零样本开放词汇语义分割方法。我们通过整合从自监督特征中提取的定位先验，局部优化了利用CLIP末层池化层的简单修改计算得到的密集MaskCLIP特征。通过这种方式，我们大幅提升了MaskCLIP的性能并生成了平滑的输出。此外，我们证明所使用的自监督特征属性可直接从CLIP特征中习得，从而仅需单次CLIP模型前向传播即可获得最优结果。我们的方法CLIP-DINOiser在推理时仅需CLIP的单次前向传播和两个轻量卷积层，无需额外监督或额外内存，并在COCO、Pascal Context、Cityscapes和ADE20k等具有挑战性的细粒度基准上达到了当前最优水平。重现我们结果的代码已开源在https://github.com/wysoczanska/clip_dinoiser。