Recent text-to-image diffusion models have demonstrated an astonishing capacity to generate high-quality images. However, researchers mainly studied the way of synthesizing images with only text prompts. While some works have explored using other modalities as conditions, considerable paired data, e.g., box/mask-image pairs, and fine-tuning time are required for nurturing models. As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world. This paper focuses on the simplest form of user-provided conditions, e.g., box or scribble. To mitigate the aforementioned problem, we propose a training-free method to control objects and contexts in the synthesized images adhering to the given spatial conditions. Specifically, three spatial constraints, i.e., Inner-Box, Outer-Box, and Corner Constraints, are designed and seamlessly integrated into the denoising step of diffusion models, requiring no additional training and massive annotated layout data. Extensive experimental results demonstrate that the proposed constraints can control what and where to present in the images while retaining the ability of Diffusion models to synthesize with high fidelity and diverse concept coverage. The code is publicly available at https://github.com/showlab/BoxDiff.

翻译：近期文本到图像扩散模型展现出生成高质量图像的惊人能力。然而，研究者主要研究了仅使用文本提示合成图像的方式。尽管已有工作探索了将其他模态作为条件，但培育模型需要大量配对数据（如框/掩码-图像对）和微调时间。由于此类配对数据获取耗时费力且局限于封闭集合，这潜在地成为开放世界应用的瓶颈。本文聚焦于用户提供条件的最简形式，例如框或涂鸦。为缓解上述问题，我们提出一种无需训练的方 法，使合成图像中的对象和上下文能够遵循给定的空间条件。具体而言，设计了三种空间约束——内框约束、外框约束及角点约束，并将其无缝集成到扩散模型的去噪步骤中，无需额外训练和海量标注布局数据。大量实验结果表明，所提出的约束既能控制图像中呈现的对象及其位置，同时保持扩散模型高保真合成与多样化概念覆盖的能力。代码已开源至 https://github.com/showlab/BoxDiff。