The pre-trained text-image discriminative models, such as CLIP, has been explored for open-vocabulary semantic segmentation with unsatisfactory results due to the loss of crucial localization information and awareness of object shapes. Recently, there has been a growing interest in expanding the application of generative models from generation tasks to semantic segmentation. These approaches utilize generative models either for generating annotated data or extracting features to facilitate semantic segmentation. This typically involves generating a considerable amount of synthetic data or requiring additional mask annotations. To this end, we uncover the potential of generative text-to-image diffusion models (e.g., Stable Diffusion) as highly efficient open-vocabulary semantic segmenters, and introduce a novel training-free approach named DiffSegmenter. The insight is that to generate realistic objects that are semantically faithful to the input text, both the complete object shapes and the corresponding semantics are implicitly learned by diffusion models. We discover that the object shapes are characterized by the self-attention maps while the semantics are indicated through the cross-attention maps produced by the denoising U-Net, forming the basis of our segmentation results.Additionally, we carefully design effective textual prompts and a category filtering mechanism to further enhance the segmentation results. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.

翻译：预训练的文本-图像判别模型（如CLIP）在开放词汇语义分割任务中因缺乏关键定位信息和目标形状感知能力而表现不佳。近年来，研究者们逐渐将生成模型的应用范围从生成任务扩展到语义分割领域。此类方法利用生成模型生成标注数据或提取特征以辅助语义分割，通常需要生成大量合成数据或额外掩码标注。为此，我们揭示了生成式文本到图像扩散模型（如Stable Diffusion）作为高效开放词汇语义分割器的潜力，并提出名为DiffSegmenter的新型训练无关方法。核心思路在于：为了生成与输入文本语义一致的真实目标，扩散模型隐式学习了完整的目标形状及其对应语义。我们发现，去噪U-Net产生的自注意力图刻画了目标形状，而交叉注意力图则指示了语义信息，这构成了我们分割结果的基础。此外，我们精心设计了有效文本提示和类别过滤机制以进一步优化分割结果。在三个基准数据集上的大量实验表明，所提出的DiffSegmenter在开放词汇语义分割中取得了令人瞩目的效果。