Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image. Our code is available at https://github.com/MACderRu/Guide-and-Rescale.

翻译：尽管大规模文本到图像生成模型近期取得了进展，但利用这些模型处理真实图像仍然是一个具有挑战性的问题。现有编辑方法的主要局限在于，它们要么无法在广泛的图像编辑任务中保持一致的性能质量，要么需要耗时的超参数调优或对扩散模型进行微调，以保留输入图像特有的外观。我们提出了一种基于改进扩散采样过程的新方法，该方法通过引导机制实现。在本工作中，我们探索了自引导技术，以保留输入图像的整体结构及其不应被编辑的局部区域外观。具体而言，我们显式引入了旨在保存源图像局部与全局结构的布局保持能量函数。此外，我们提出了一种噪声重缩放机制，通过在生成过程中平衡无分类器引导与我们提出的引导器的范数，从而保持噪声分布。这种引导方法无需对扩散模型进行微调或执行精确的反转过程。因此，所提出的方法提供了一种快速且高质量的编辑机制。在我们的实验中，通过人工评估与定量分析，我们证明了所提出的方法能够生成更受人类偏好的预期编辑效果，并在编辑质量与原始图像保持之间实现了更好的权衡。我们的代码发布于 https://github.com/MACderRu/Guide-and-Rescale。