Weakly supervised semantic segmentation (WSSS) with image-level labels is a challenging task. Mainstream approaches follow a multi-stage framework and suffer from high training costs. In this paper, we explore the potential of Contrastive Language-Image Pre-training models (CLIP) to localize different categories with only image-level labels and without further training. To efficiently generate high-quality segmentation masks from CLIP, we propose a novel WSSS framework called CLIP-ES. Our framework improves all three stages of WSSS with special designs for CLIP: 1) We introduce the softmax function into GradCAM and exploit the zero-shot ability of CLIP to suppress the confusion caused by non-target classes and backgrounds. Meanwhile, to take full advantage of CLIP, we re-explore text inputs under the WSSS setting and customize two text-driven strategies: sharpness-based prompt selection and synonym fusion. 2) To simplify the stage of CAM refinement, we propose a real-time class-aware attention-based affinity (CAA) module based on the inherent multi-head self-attention (MHSA) in CLIP-ViTs. 3) When training the final segmentation model with the masks generated by CLIP, we introduced a confidence-guided loss (CGL) focus on confident regions. Our CLIP-ES achieves SOTA performance on Pascal VOC 2012 and MS COCO 2014 while only taking 10% time of previous methods for the pseudo mask generation. Code is available at https://github.com/linyq2117/CLIP-ES.

翻译：弱监督语义分割（WSSS）仅利用图像级标签完成分割任务极具挑战性。主流方法遵循多阶段框架且面临高昂训练成本。本文探索对比语言-图像预训练模型（CLIP）在仅使用图像级标签且无需额外训练的条件下定位不同类别的潜力。为高效生成高质量分割掩码，我们提出名为CLIP-ES的新型WSSS框架，该框架针对CLIP特性对WSSS的三个阶段进行特殊设计：1）将softmax函数引入GradCAM，并利用CLIP的零样本能力抑制非目标类别与背景造成的混淆。同时为充分挖掘CLIP潜力，我们重新探索WSSS设定下的文本输入，定制两种文本驱动策略：基于锐度的提示选择与同义词融合；2）为简化CAM细化阶段，基于CLIP-ViTs固有的多头自注意力（MHSA）提出实时类感知注意力亲和模块（CAA）；3）在使用CLIP生成的掩码训练最终分割模型时，引入聚焦于置信区域的置信度引导损失（CGL）。我们的CLIP-ES在Pascal VOC 2012和MS COCO 2014上达到最优性能（SOTA），而伪掩码生成时间仅为先前方法的10%。代码开源于https://github.com/linyq2117/CLIP-ES。