Current text-to-image editing models often encounter challenges with smoothly manipulating multiple attributes using a single instruction. Taking inspiration from the Chain-of-Thought prompting technique utilized in language models, we present an innovative concept known as Chain-of-Instruct Editing (CoIE), which enhances the capabilities of these models through step-by-step editing using a series of instructions. In particular, in the context of face manipulation, we leverage the contextual learning abilities of a pretrained Large Language Model (LLM), such as GPT-4, to generate a sequence of instructions from the original input, utilizing a purpose-designed 1-shot template. To further improve the precision of each editing step, we conduct fine-tuning on the editing models using our self-constructed instruction-guided face editing dataset, Instruct-CelebA. And additionally, we incorporate a super-resolution module to mitigate the adverse effects of editability and quality degradation. Experimental results across various challenging cases confirm the significant boost in multi-attribute facial image manipulation using chain-of-instruct editing. This is evident in enhanced editing success rates, measured by CLIPSim and Coverage metrics, improved by 17.86% and 85.45% respectively, and heightened controllability indicated by Preserve L1 and Quality metrics, improved by 11.58% and 4.93% respectively.
翻译:当前文本到图像编辑模型在通过单条指令平滑操控多个属性时往往面临挑战。受语言模型中思维链提示技术启发,我们提出一种称为"指令链编辑"(Chain-of-Instruct Editing,CoIE)的创新概念,通过逐步执行一系列指令来增强模型的多属性编辑能力。具体而言,在人脸操控场景中,我们利用预训练大语言模型(如GPT-4)的上下文学习能力,基于专门设计的1样本模板,从原始输入中生成指令序列。为提升每个编辑步骤的精度,我们使用自建的指令引导人脸编辑数据集Instruct-CelebA对编辑模型进行微调。此外,我们引入超分辨率模块以缓解可编辑性与质量退化带来的不利影响。针对多种挑战性案例的实验结果表明,指令链编辑显著提升了多属性人脸图像操控性能:由CLIPSim和覆盖度指标衡量的编辑成功率分别提升17.86%和85.45%,由保留度L1和质量指标体现的可控性分别改善11.58%和4.93%。