Recent text-to-image diffusion models have demonstrated an astonishing capacity to generate high-quality images. However, researchers mainly studied the way of synthesizing images with only text prompts. While some works have explored using other modalities as conditions, considerable paired data, e.g., box/mask-image pairs, and fine-tuning time are required for nurturing models. As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world. This paper focuses on the simplest form of user-provided conditions, e.g., box or scribble. To mitigate the aforementioned problem, we propose a training-free method to control objects and contexts in the synthesized images adhering to the given spatial conditions. Specifically, three spatial constraints, i.e., Inner-Box, Outer-Box, and Corner Constraints, are designed and seamlessly integrated into the denoising step of diffusion models, requiring no additional training and massive annotated layout data. Extensive results show that the proposed constraints can control what and where to present in the images while retaining the ability of the Stable Diffusion model to synthesize with high fidelity and diverse concept coverage. The code is publicly available at https://github.com/Sierkinhane/BoxDiff.

翻译：近期文本到图像扩散模型在生成高质量图像方面展现出惊人能力。然而，研究者主要聚焦于仅通过文本提示合成图像的方式。虽然已有工作探索将其他模态作为条件输入，但培育模型需要大量配对数据（如框/掩码-图像对）及微调时间。由于此类配对数据的获取耗时费力且局限于封闭集合，这或将成为开放世界应用中的瓶颈。本文聚焦于用户提供的最简条件形式（如框或涂鸦）。为缓解上述问题，我们提出一种免训练方法，使合成图像中的物体与背景能遵循给定空间条件。具体而言，我们设计了三种空间约束（内框约束、外框约束与角点约束），并将其无缝整合至扩散模型的去噪步骤中，无需额外训练和海量标注布局数据。大量实验表明，所提约束可控制图像中呈现的内容及其位置，同时保持Stable Diffusion模型高保真合成与广泛概念覆盖的能力。代码已公开于https://github.com/Sierkinhane/BoxDiff。