Diffusion models have exhibited impressive prowess in the text-to-image task. Recent methods add image-level controls, e.g., edge and depth maps, to manipulate the generation process together with text prompts to obtain desired images. This controlling process is globally operated on the entire image, which limits the flexibility of control regions. In this paper, we introduce a new simple yet practical task setting: local control. It focuses on controlling specific local areas according to user-defined image conditions, where the rest areas are only conditioned by the original text prompt. This manner allows the users to flexibly control the image generation in a fine-grained way. However, it is non-trivial to achieve this goal. The naive manner of directly adding local conditions may lead to the local control dominance problem. To mitigate this problem, we propose a training-free method that leverages the updates of noised latents and parameters in the cross-attention map during the denosing process to promote concept generation in non-control areas. Moreover, we use feature mask constraints to mitigate the degradation of synthesized image quality caused by information differences inside and outside the local control area. Extensive experiments demonstrate that our method can synthesize high-quality images to the prompt under local control conditions. Code is available at https://github.com/YibooZhao/Local-Control.

翻译：扩散模型在文本到图像任务中展现出显著的能力。近期方法通过添加图像级条件（如边缘图、深度图）以及文本提示来操纵生成过程，以获得期望的图像。这种控制过程是对整幅图像进行全局操作的，限制了控制区域的灵活性。本文提出了一种新颖、简单且实用的任务设置：局部控制。该设置侧重于根据用户定义的图像条件控制特定的局部区域，其余区域仅受原始文本提示的约束。这种方式使用户能够以细粒度的方式灵活控制图像生成。然而，实现这一目标并非易事。直接添加局部条件的朴素方法可能导致“局部控制主导”问题。为缓解该问题，我们提出了一种无需训练的方法，利用去噪过程中噪声潜向量和交叉注意力图参数的更新，促进非控制区域的概念生成。此外，我们采用特征掩码约束来减轻因局部控制区域内外信息差异导致的合成图像质量退化。大量实验表明，我们的方法能够在局部控制条件下生成与提示一致的高质量图像。代码详见 https://github.com/YibooZhao/Local-Control。