Pre-trained vision-language models, e.g., CLIP, have been successfully applied to zero-shot semantic segmentation. Existing CLIP-based approaches primarily utilize visual features from the last layer to align with text embeddings, while they neglect the crucial information in intermediate layers that contain rich object details. However, we find that directly aggregating the multi-level visual features weakens the zero-shot ability for novel classes. The large differences between the visual features from different layers make these features hard to align well with the text embeddings. We resolve this problem by introducing a series of independent decoders to align the multi-level visual features with the text embeddings in a cascaded way, forming a novel but simple framework named Cascade-CLIP. Our Cascade-CLIP is flexible and can be easily applied to existing zero-shot semantic segmentation methods. Experimental results show that our simple Cascade-CLIP achieves superior zero-shot performance on segmentation benchmarks, like COCO-Stuff, Pascal-VOC, and Pascal-Context. Our code is available at: https://github.com/HVision-NKU/Cascade-CLIP

翻译：预训练的视觉-语言模型（如CLIP）已成功应用于零样本语义分割。现有的基于CLIP的方法主要利用最后一层的视觉特征与文本嵌入进行对齐，而忽略了中间层包含丰富物体细节的关键信息。然而，我们发现直接聚合多层级视觉特征会削弱模型对新类别的零样本能力。不同层级视觉特征之间的巨大差异使得这些特征难以与文本嵌入良好对齐。我们通过引入一系列独立解码器，以级联方式将多层级视觉特征与文本嵌入对齐，从而解决了这一问题，形成了一个新颖而简洁的框架，命名为Cascade-CLIP。我们的Cascade-CLIP具有灵活性，可轻松应用于现有的零样本语义分割方法。实验结果表明，我们简洁的Cascade-CLIP在COCO-Stuff、Pascal-VOC和Pascal-Context等分割基准数据集上实现了优异的零样本性能。代码已开源：https://github.com/HVision-NKU/Cascade-CLIP