Recent text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images given text-prompts as input. However, these models fail to convey appropriate spatial composition specified by a layout instruction. In this work, we probe into zero-shot grounded T2I generation with diffusion models, that is, generating images corresponding to the input layout information without training auxiliary modules or finetuning diffusion models. We propose a Region and Boundary (R&B) aware cross-attention guidance approach that gradually modulates the attention maps of diffusion model during generative process, and assists the model to synthesize images (1) with high fidelity, (2) highly compatible with textual input, and (3) interpreting layout instructions accurately. Specifically, we leverage the discrete sampling to bridge the gap between consecutive attention maps and discrete layout constraints, and design a region-aware loss to refine the generative layout during diffusion process. We further propose a boundary-aware loss to strengthen object discriminability within the corresponding regions. Experimental results show that our method outperforms existing state-of-the-art zero-shot grounded T2I generation methods by a large margin both qualitatively and quantitatively on several benchmarks.

翻译：摘要：近年来，文本到图像扩散模型在根据输入文本提示生成高质量图像方面取得了显著进展。然而，这些模型难以准确传达布局指令所要求的空间组合关系。本文探索了基于扩散模型的零样本空间约束文本到图像生成方法，即在不训练辅助模块或微调扩散模型的情况下，根据输入布局信息生成对应图像。我们提出了一种区域与边界感知的交叉注意力引导方法，该方法在生成过程中逐步调节扩散模型的注意力图，辅助模型生成：(1)高保真度、(2)与文本输入高度兼容、(3)精确解释布局指令的图像。具体而言，我们利用离散采样弥合连续注意力图与离散布局约束之间的差异，并设计区域感知损失函数以优化扩散过程中的生成布局。进一步提出边界感知损失函数以增强对应区域内物体的可区分性。实验结果表明，在多个基准数据集上，本方法在定性与定量指标上均显著优于现有最先进的零样本空间约束文本到图像生成方法。