Despite the recent advances in large-scale diffusion models, little progress has been made on the layout-to-image (L2I) synthesis task. Current L2I models either suffer from poor editability via text or weak alignment between the generated image and the input layout. This limits their usability in practice. To mitigate this, we propose to integrate adversarial supervision into the conventional training pipeline of L2I diffusion models (ALDM). Specifically, we employ a segmentation-based discriminator which provides explicit feedback to the diffusion generator on the pixel-level alignment between the denoised image and the input layout. To encourage consistent adherence to the input layout over the sampling steps, we further introduce the multistep unrolling strategy. Instead of looking at a single timestep, we unroll a few steps recursively to imitate the inference process, and ask the discriminator to assess the alignment of denoised images with the layout over a certain time window. Our experiments show that ALDM enables layout faithfulness of the generated images, while allowing broad editability via text prompts. Moreover, we showcase its usefulness for practical applications: by synthesizing target distribution samples via text control, we improve domain generalization of semantic segmentation models by a large margin (~12 mIoU points).

翻译：尽管近期大规模扩散模型取得了进展，但布局到图像（L2I）合成任务的研究进展甚微。当前的L2I模型要么难以通过文本进行有效编辑，要么生成图像与输入布局之间的对齐性较弱，这限制了其实用性。为此，我们提出将对抗性监督融入L2I扩散模型（ALDM）的传统训练流程。具体而言，我们采用基于分割的判别器，该判别器向扩散生成器提供关于去噪图像与输入布局之间像素级对齐的显式反馈。为促进采样步骤中对输入布局的一致遵循，我们进一步引入多步展开策略。不同于仅关注单一时间步，我们递归地展开若干步以模拟推理过程，并要求判别器在特定时间窗口内评估去噪图像与布局的对齐程度。实验表明，ALDM在保证生成图像布局忠实性的同时，允许通过文本提示进行广泛编辑。此外，我们展示了其实际应用价值：通过文本控制合成目标分布样本，我们将语义分割模型的领域泛化性能大幅提升（约12个mIoU点）。