Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a two-stage scheme. The general idea is to first generate class-agnostic region proposals and then feed the cropped proposal regions to CLIP to utilize its image-level zero-shot classification capability. While effective, such a scheme requires two image encoders, one for proposal generation and one for CLIP, leading to a complicated pipeline and high computational cost. In this work, we pursue a simpler-and-efficient one-stage solution that directly extends CLIP's zero-shot prediction capability from image to pixel level. Our investigation starts with a straightforward extension as our baseline that generates semantic masks by comparing the similarity between text and patch embeddings extracted from CLIP. However, such a paradigm could heavily overfit the seen classes and fail to generalize to unseen classes. To handle this issue, we propose three simple-but-effective designs and figure out that they can significantly retain the inherent zero-shot capacity of CLIP and improve pixel-level generalization ability. Incorporating those modifications leads to an efficient zero-shot semantic segmentation system called ZegCLIP. Through extensive experiments on three public benchmarks, ZegCLIP demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both "inductive" and "transductive" zero-shot settings. In addition, compared with the two-stage method, our one-stage ZegCLIP achieves a speedup of about 5 times faster during inference. We release the code at https://github.com/ZiqinZhou66/ZegCLIP.git.

翻译：近期，CLIP通过两阶段方案被应用于像素级零样本学习任务。其核心思路是先生成与类别无关的区域提议，再将裁剪出的提议区域输入CLIP，以利用其图像级零样本分类能力。尽管有效，但该方案需要两个图像编码器（一个用于提议生成，一个用于CLIP），导致流程复杂且计算成本高昂。本文探索一种更简单高效的单阶段解决方案，直接扩展CLIP的零样本预测能力从图像级至像素级。我们的研究以一项直接扩展作为基线：通过比较从CLIP提取的文本嵌入与图像补丁嵌入之间的相似度来生成语义掩码。然而，该范式可能严重过拟合可见类别，难以泛化至未见类别。为解决此问题，我们提出三项简单而有效的设计，发现它们能显著保留CLIP固有的零样本能力并提升像素级泛化能力。整合这些改进后，我们构建了一套高效的零样本语义分割系统——ZegCLIP。在三个公开基准上的大量实验表明，ZegCLIP在"归纳式"和"直推式"零样本设定下均以显著优势超越当前最优方法。此外，相较于两阶段方法，我们的单阶段ZegCLIP在推理时实现了约5倍加速。代码已开源至https://github.com/ZiqinZhou66/ZegCLIP.git。