This paper addresses text-supervised semantic segmentation, aiming to learn a model capable of segmenting arbitrary visual concepts within images by using only image-text pairs without dense annotations. Existing methods have demonstrated that contrastive learning on image-text pairs effectively aligns visual segments with the meanings of texts. We notice that there is a discrepancy between text alignment and semantic segmentation: A text often consists of multiple semantic concepts, whereas semantic segmentation strives to create semantically homogeneous segments. To address this issue, we propose a novel framework, Image-Text Co-Decomposition (CoDe), where the paired image and text are jointly decomposed into a set of image regions and a set of word segments, respectively, and contrastive learning is developed to enforce region-word alignment. To work with a vision-language model, we present a prompt learning mechanism that derives an extra representation to highlight an image segment or a word segment of interest, with which more effective features can be extracted from that segment. Comprehensive experimental results demonstrate that our method performs favorably against existing text-supervised semantic segmentation methods on six benchmark datasets.

翻译：本文研究文本监督语义分割问题，旨在通过仅使用图像-文本对（无需密集标注）来训练能够分割图像中任意视觉概念的模型。现有方法已证明，对图像-文本对进行对比学习能有效对齐视觉片段与文本语义。我们发现文本对齐与语义分割之间存在差异：文本通常包含多个语义概念，而语义分割致力于生成语义同质的片段。为解决该问题，我们提出新型框架——图像-文本联合分解（CoDe），其中配对的图像与文本分别被分解为一系列图像区域和一组词片段，并设计对比学习来实现区域-词对齐。为适配视觉-语言模型，我们提出提示学习机制，通过推导额外表示来突出感兴趣图像片段或词片段，从而从该片段中提取更有效的特征。综合实验结果表明，在六个基准数据集上，我们的方法性能优于现有文本监督语义分割方法。