Recent research has explored the utilization of pre-trained text-image discriminative models, such as CLIP, to tackle the challenges associated with open-vocabulary semantic segmentation. However, it is worth noting that the alignment process based on contrastive learning employed by these models may unintentionally result in the loss of crucial localization information and object completeness, which are essential for achieving accurate semantic segmentation. More recently, there has been an emerging interest in extending the application of diffusion models beyond text-to-image generation tasks, particularly in the domain of semantic segmentation. These approaches utilize diffusion models either for generating annotated data or for extracting features to facilitate semantic segmentation. This typically involves training segmentation models by generating a considerable amount of synthetic data or incorporating additional mask annotations. To this end, we uncover the potential of generative text-to-image conditional diffusion models as highly efficient open-vocabulary semantic segmenters, and introduce a novel training-free approach named DiffSegmenter. Specifically, by feeding an input image and candidate classes into an off-the-shelf pre-trained conditional latent diffusion model, the cross-attention maps produced by the denoising U-Net are directly used as segmentation scores, which are further refined and completed by the followed self-attention maps. Additionally, we carefully design effective textual prompts and a category filtering mechanism to further enhance the segmentation results. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.

翻译：最近的研究探索了利用预训练的文本-图像判别模型（如CLIP）来解决开集语义分割的挑战。然而值得注意的是，这些模型基于对比学习的对齐过程可能无意中导致关键定位信息和目标完整性的丢失，而这两者对于实现精确的语义分割至关重要。近期，将扩散模型的应用拓展到文本到图像生成任务之外（特别是在语义分割领域）的研究兴趣日益增长。这些方法利用扩散模型生成标注数据或提取特征来辅助语义分割，通常需要生成大量合成数据或引入额外的掩码标注来训练分割模型。为此，我们揭示了生成式文本到图像条件扩散模型作为高效开集语义分割器的潜力，并提出了一种名为DiffSegmenter的新型无训练方法。具体而言，通过将输入图像和候选类别输入现成的预训练条件隐扩散模型，由去噪U-Net生成的交叉注意力图可直接作为分割分数，并通过后续的自注意力图进一步优化和完善。此外，我们精心设计了有效的文本提示和类别过滤机制以进一步提升分割效果。在三个基准数据集上的广泛实验表明，所提出的DiffSegmenter在开集语义分割任务上取得了令人瞩目的成果。