The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both worlds and propose an open-vocabulary semantic segmentation method, which does not require any annotations. We propose to locally improve dense MaskCLIP features, which are computed with a simple modification of CLIP's last pooling layer, by integrating localization priors extracted from self-supervised features. By doing so, we greatly improve the performance of MaskCLIP and produce smooth outputs. Moreover, we show that the used self-supervised feature properties can directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference, no extra supervision nor extra memory and reaches state-of-the-art results on challenging and fine-grained benchmarks such as COCO, Pascal Context, Cityscapes and ADE20k. The code to reproduce our results is available at https://github.com/wysoczanska/clip_dinoiser.

翻译：流行的CLIP模型通过与任意文本提示的无缝交互展现出令人印象深刻的零样本能力。然而，由于其缺乏空间感知能力，若不进行额外的微调步骤（该步骤通常使用标注且可能抑制其原有的开放词汇属性），它难以适用于密集计算机视觉任务（如语义分割）。与此同时，自监督表示方法无需人工标注或显式监督即可展现出良好的定位特性。在这项工作中，我们取两者之长，提出了一种无需任何标注的开放词汇语义分割方法。我们通过整合从自监督特征中提取的定位先验，局部改进了密集MaskCLIP特征——这些特征通过对CLIP最后一个池化层进行简单修改计算得到。通过此举，我们显著提升了MaskCLIP的性能并生成平滑的输出。此外，我们证明所使用的自监督特征属性可以直接从CLIP特征中学习得到。我们的方法CLIP-DINOiser在推理时仅需对CLIP进行一次前向传播和两个轻量级卷积层，无需额外监督或额外内存，并且在COCO、Pascal Context、Cityscapes和ADE20k等具有挑战性的细粒度基准上达到了最先进水平。复现我们结果的代码可在 https://github.com/wysoczanska/clip_dinoiser 获取。