Recently, diffusion models have increasingly demonstrated their capabilities in vision understanding. By leveraging prompt-based learning to construct sentences, these models have shown proficiency in classification and visual grounding tasks. However, existing approaches primarily showcase their ability to perform sentence-level localization, leaving the potential for leveraging contextual information for phrase-level understanding largely unexplored. In this paper, we utilize Panoptic Narrative Grounding (PNG) as a proxy task to investigate this capability further. PNG aims to segment object instances mentioned by multiple noun phrases within a given narrative text. Specifically, we introduce the DiffPNG framework, a straightforward yet effective approach that fully capitalizes on the diffusion's architecture for segmentation by decomposing the process into a sequence of localization, segmentation, and refinement steps. The framework initially identifies anchor points using cross-attention mechanisms and subsequently performs segmentation with self-attention to achieve zero-shot PNG. Moreover, we introduce a refinement module based on SAM to enhance the quality of the segmentation masks. Our extensive experiments on the PNG dataset demonstrate that DiffPNG achieves strong performance in the zero-shot PNG task setting, conclusively proving the diffusion model's capability for context-aware, phrase-level understanding. Source code is available at \url{https://github.com/nini0919/DiffPNG}.

翻译：近年来，扩散模型在视觉理解任务中日益展现出其强大能力。通过利用基于提示的学习来构建句子，这些模型已在分类和视觉定位任务中表现出色。然而，现有方法主要展示了其在句子级定位方面的能力，而利用上下文信息进行短语级理解的潜力在很大程度上尚未得到充分探索。本文采用全景叙事定位（PNG）作为代理任务来进一步探究这种能力。PNG旨在分割给定叙事文本中由多个名词短语提及的物体实例。具体而言，我们提出了DiffPNG框架，这是一种简单而有效的方法，通过将分割过程分解为定位、分割和细化三个步骤，充分利用了扩散模型架构进行分割。该框架首先利用交叉注意力机制识别锚点，随后通过自注意力进行分割，从而实现零样本PNG。此外，我们引入了一个基于SAM的细化模块来提升分割掩码的质量。我们在PNG数据集上进行的大量实验表明，DiffPNG在零样本PNG任务设置中取得了强劲的性能，充分证明了扩散模型具备上下文感知的短语级理解能力。源代码发布于\url{https://github.com/nini0919/DiffPNG}。