Recent text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images given text-prompts as input. However, these models fail to convey appropriate spatial composition specified by a layout instruction. In this work, we probe into zero-shot grounded T2I generation with diffusion models, that is, generating images corresponding to the input layout information without training auxiliary modules or finetuning diffusion models. We propose a Region and Boundary (R&B) aware cross-attention guidance approach that gradually modulates the attention maps of diffusion model during generative process, and assists the model to synthesize images (1) with high fidelity, (2) highly compatible with textual input, and (3) interpreting layout instructions accurately. Specifically, we leverage the discrete sampling to bridge the gap between consecutive attention maps and discrete layout constraints, and design a region-aware loss to refine the generative layout during diffusion process. We further propose a boundary-aware loss to strengthen object discriminability within the corresponding regions. Experimental results show that our method outperforms existing state-of-the-art zero-shot grounded T2I generation methods by a large margin both qualitatively and quantitatively on several benchmarks.

翻译：近年来，文本到图像（T2I）扩散模型在根据文本提示生成高质量图像方面取得了显著进展。然而，这些模型无法准确传达由布局指令指定的空间组合。本文探索了基于扩散模型的零样本接地T2I生成方法，即无需训练辅助模块或微调扩散模型，即可生成与输入布局信息相对应的图像。我们提出了一种区域与边界（R&B）感知的交叉注意力引导方法，该方法在生成过程中逐步调节扩散模型的注意力图，帮助模型合成（1）高保真度、（2）与文本输入高度兼容且（3）准确解读布局指令的图像。具体而言，我们利用离散采样来弥合连续注意力图与离散布局约束之间的差距，并设计了一种区域感知损失以在扩散过程中优化生成布局。进一步，我们提出了一种边界感知损失以增强对应区域内物体的区分性。实验结果表明，在多个基准测试中，我们的方法在定性和定量上均大幅超越现有最先进的零样本接地T2I生成方法。