Diffusion models have revolted the field of text-to-image generation recently. The unique way of fusing text and image information contributes to their remarkable capability of generating highly text-related images. From another perspective, these generative models imply clues about the precise correlation between words and pixels. In this work, a simple but effective method is proposed to utilize the attention mechanism in the denoising network of text-to-image diffusion models. Without re-training nor inference-time optimization, the semantic grounding of phrases can be attained directly. We evaluate our method on Pascal VOC 2012 and Microsoft COCO 2014 under weakly-supervised semantic segmentation setting and our method achieves superior performance to prior methods. In addition, the acquired word-pixel correlation is found to be generalizable for the learned text embedding of customized generation methods, requiring only a few modifications. To validate our discovery, we introduce a new practical task called "personalized referring image segmentation" with a new dataset. Experiments in various situations demonstrate the advantages of our method compared to strong baselines on this task. In summary, our work reveals a novel way to extract the rich multi-modal knowledge hidden in diffusion models for segmentation.

翻译：扩散模型最近彻底改变了文本到图像生成领域。其融合文本与图像信息的独特方式，使其具备生成高度相关文本图像的卓越能力。从另一角度看，这些生成模型揭示了词与像素之间精确关联的线索。本文提出一种简单而有效的方法，利用文本到图像扩散模型去噪网络中的注意力机制。无需重新训练或推理时优化，即可直接获取短语的语义定位。我们在Pascal VOC 2012和Microsoft COCO 2014数据集上，基于弱监督语义分割设置进行评估，该方法取得了优于先前方法的性能。此外，研究发现，所获取的词-像素关联可泛化至定制化生成方法的学习文本嵌入，仅需少量修改即可实现。为验证这一发现，我们引入一项名为“个性化指代图像分割”的新实用任务，并构建了新数据集。多种场景下的实验表明，我们的方法在此任务上相较于强基线具有优势。总之，本研究揭示了一种从扩散模型中提取丰富多模态知识用于分割的新途径。