Class Activation Map (CAM) has emerged as a popular tool for weakly supervised semantic segmentation (WSSS), allowing the localization of object regions in an image using only image-level labels. However, existing CAM methods suffer from under-activation of target object regions and false-activation of background regions due to the fact that a lack of detailed supervision can hinder the model's ability to understand the image as a whole. In this paper, we propose a novel Question-Answer Cross-Language-Image Matching framework for WSSS (QA-CLIMS), leveraging the vision-language foundation model to maximize the text-based understanding of images and guide the generation of activation maps. First, a series of carefully designed questions are posed to the VQA (Visual Question Answering) model with Question-Answer Prompt Engineering (QAPE) to generate a corpus of both foreground target objects and backgrounds that are adaptive to query images. We then employ contrastive learning in a Region Image Text Contrastive (RITC) network to compare the obtained foreground and background regions with the generated corpus. Our approach exploits the rich textual information from the open vocabulary as additional supervision, enabling the model to generate high-quality CAMs with a more complete object region and reduce false-activation of background regions. We conduct extensive analysis to validate the proposed method and show that our approach performs state-of-the-art on both PASCAL VOC 2012 and MS COCO datasets. Code is available at: https://github.com/CVI-SZU/QA-CLIMS

翻译：类别激活图（CAM）已成为弱监督语义分割（WSSS）的流行工具，能够仅利用图像级标签定位图像中的目标物体区域。然而，现有CAM方法因缺乏细粒度监督而难以让模型充分理解图像整体，导致目标区域激活不足和背景区域误激活。本文提出一种新颖的基于问答跨语言图像匹配框架QA-CLIMS，利用视觉-语言基础模型最大化图像基于文本的理解，并指导激活图的生成。首先，通过问答提示工程（QAPE）对视觉问答（VQA）模型提出一系列精心设计的问题，生成适应查询图像的前景目标物体和背景语料库。随后，我们在区域图像文本对比（RITC）网络中采用对比学习，将获取的前景和背景区域与生成的语料库进行比对。该方法利用开放词库中丰富的文本信息作为额外监督，使模型生成具有更完整目标区域的高质量CAM，并减少背景区域的误激活。我们通过广泛分析验证了所提方法的有效性，并在PASCAL VOC 2012和MS COCO数据集上取得了最先进的性能。代码地址：https://github.com/CVI-SZU/QA-CLIMS