Text guided image editing on real images given only the image and the target text prompt as inputs, is a very general and challenging problem, which requires the editing model to reason by itself which part of the image should be edited, to preserve the characteristics of original image, and also to perform complicated non-rigid editing. Previous fine-tuning based solutions are time-consuming and vulnerable to overfitting, limiting their editing capabilities. To tackle these issues, we design a novel text guided image editing method, Forgedit. First, we propose a novel fine-tuning framework which learns to reconstruct the given image in less than one minute by vision language joint learning. Then we introduce vector subtraction and vector projection to explore the proper text embedding for editing. We also find a general property of UNet structures in Diffusion Models and inspired by such a finding, we design forgetting strategies to diminish the fatal overfitting issues and significantly boost the editing abilities of Diffusion Models. Our method, Forgedit, implemented with Stable Diffusion, achieves new state-of-the-art results on the challenging text guided image editing benchmark TEdBench, surpassing the previous SOTA method Imagic with Imagen, in terms of both CLIP score and LPIPS score. Codes are available at https://github.com/witcherofresearch/Forgedit.

翻译：文本引导的真实图像编辑是一项非常通用且具有挑战性的任务，其输入仅为图像与目标文本提示，要求编辑模型自主推理应编辑图像的哪些部分，同时保留原始图像的特征，并完成复杂的非刚性编辑。以往基于微调的方法耗时且容易过拟合，限制了其编辑能力。为解决这些问题，我们设计了一种新颖的文本引导图像编辑方法——Forgedit。首先，我们提出了一种新型微调框架，通过视觉语言联合学习在不到一分钟内学会重构给定图像。随后引入向量减法与向量投影以探索适合编辑的文本嵌入。我们还发现了扩散模型中UNet结构的一个通用性质，并受此启发设计了遗忘策略，以消除致命的过拟合问题，显著提升扩散模型的编辑能力。我们的方法Forgedit基于Stable Diffusion实现，在具有挑战性的文本引导图像编辑基准TEdBench上取得了新的最优结果，在CLIP得分与LPIPS得分上均超越了此前基于Imagen的最优方法Imagic。代码见https://github.com/witcherofresearch/Forgedit。