Diffusion models are a powerful class of generative models capable of producing high-quality images from pure noise. In particular, conditional diffusion models allow one to specify the contents of the desired image using a simple text prompt. Conditioning on a text prompt alone, however, does not allow for fine-grained control over the composition and layout of the final image, which instead depends closely on the initial noise distribution. While most methods which introduce spatial constraints (e.g., bounding boxes) require fine-tuning, a smaller and more recent subset of these methods are training-free. They are applicable whenever the prompt influences the model through an attention mechanism, and generally fall into one of two categories. The first entails modifying the cross-attention maps of specific tokens directly to enhance the signal in certain regions of the image. The second works by defining a loss function over the cross-attention maps, and using the gradient of this loss to guide the latent. While previous work explores these as alternative strategies, we provide an interpretation for these methods which highlights their complimentary features, and demonstrate that it is possible to obtain superior performance when both methods are used in concert.

翻译：扩散模型是一类强大的生成模型，能够从纯噪声生成高质量图像。特别是条件扩散模型允许用户通过简单的文本提示来指定期望图像的内容。然而，仅依赖文本提示进行条件控制无法实现对最终图像构图与布局的精细调控，这些特性反而紧密依赖于初始噪声分布。虽然大多数引入空间约束（例如边界框）的方法需要进行微调，但其中规模较小且更近期的子集属于无需训练的方法。这些方法适用于提示通过注意力机制影响模型的任何情况，通常可分为两类。第一类直接修改特定标记的交叉注意力图，以增强图像特定区域的信号。第二类通过在交叉注意力图上定义损失函数，并利用该损失的梯度来引导潜变量。尽管先前研究将这些方法视为替代策略，我们提供了一种解释框架，突显了它们的互补特性，并证明当两种方法协同使用时能够获得更优的性能。