Recently, text-to-image diffusion models have shown remarkable capabilities in creating realistic images from natural language prompts. However, few works have explored using these models for semantic localization or grounding. In this work, we explore how an off-the-shelf text-to-image diffusion model, trained without exposure to localization information, can ground various semantic phrases without segmentation-specific re-training. We introduce an inference time optimization process capable of generating segmentation masks conditioned on natural language prompts. Our proposal, Peekaboo, is a first-of-its-kind zero-shot, open-vocabulary, unsupervised semantic grounding technique leveraging diffusion models without any training. We evaluate Peekaboo on the Pascal VOC dataset for unsupervised semantic segmentation and the RefCOCO dataset for referring segmentation, showing results competitive with promising results. We also demonstrate how Peekaboo can be used to generate images with transparency, even though the underlying diffusion model was only trained on RGB images - which to our knowledge we are the first to attempt. Please see our project page, including our code: https://ryanndagreat.github.io/peekaboo

翻译：近期，文本到图像扩散模型在根据自然语言提示生成逼真图像方面展现出卓越能力。然而，目前鲜有研究探索如何将这些模型用于语义定位或接地。本文研究了未经定位信息训练的现成文本到图像扩散模型，如何在无需特定分割重训练的情况下，实现对各类语义短语的接地。我们提出了一种推理时优化过程，能够根据自然语言提示生成分割掩码。我们的方法Peekaboo是一种开创性的零样本、开放词汇、无监督语义接地技术，无需任何训练即利用扩散模型。我们在Pascal VOC数据集上评估了Peekaboo的无监督语义分割性能，并在RefCOCO数据集上评估了其指代分割能力，结果显示其具有良好的竞争性表现。我们还展示了Peekaboo如何用于生成带透明度的图像——尽管底层扩散模型仅基于RGB图像训练（据我们所知，这是首次尝试）。请参阅我们的项目页面（含代码）：https://ryanndagreat.github.io/peekaboo