Recent text-to-image diffusion models have demonstrated an astonishing capacity to generate high-quality images. However, researchers mainly studied the way of synthesizing images with only text prompts. While some works have explored using other modalities as conditions, considerable paired data, e.g., box/mask-image pairs, and fine-tuning time are required for nurturing models. As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world. This paper focuses on the simplest form of user-provided conditions, e.g., box or scribble. To mitigate the aforementioned problem, we propose a training-free method to control objects and contexts in the synthesized images adhering to the given spatial conditions. Specifically, three spatial constraints, i.e., Inner-Box, Outer-Box, and Corner Constraints, are designed and seamlessly integrated into the denoising step of diffusion models, requiring no additional training and massive annotated layout data. Extensive results show that the proposed constraints can control what and where to present in the images while retaining the ability of the Stable Diffusion model to synthesize with high fidelity and diverse concept coverage. The code is publicly available at https://github.com/Sierkinhane/BoxDiff.

翻译：近年来，文本到图像扩散模型展现出了生成高质量图像的惊人能力。然而，现有研究主要集中于仅通过文本提示合成图像的方式。尽管部分工作探索了将其他模态作为条件进行生成，但这类方法需要大量配对数据（如框/掩码-图像对）及微调时间以培育模型。由于此类配对数据获取耗时费力且局限于封闭集合，这很可能成为开放世界应用的瓶颈。本文聚焦于用户提供条件的最简形式（如框或涂鸦）。为缓解上述问题，我们提出一种无需训练的方法，使合成图像中的物体和背景遵循给定的空间条件。具体而言，我们设计了三种空间约束——内框约束、外框约束和角落约束，并将其无缝集成到扩散模型的去噪步骤中，无需额外训练和大量标注布局数据。大量实验表明，所提出的约束不仅能控制图像中呈现的内容及其位置，同时保留了Stable Diffusion模型高保真合成和多样化概念覆盖的能力。代码开源地址：https://github.com/Sierkinhane/BoxDiff