Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method DPL, based on the publicly available Stable Diffusion, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

翻译：大规模文本到图像生成模型已成为生成式AI领域的突破性进展，扩散模型展现出根据输入文本提示合成逼真图像的惊人能力。图像编辑研究的目标是通过修改文本提示让用户控制生成图像。当前图像编辑技术容易导致目标区域外的意外修改，例如背景或与目标对象具有语义或视觉关联的干扰对象。我们的实验发现，不准确的跨注意力映射是此问题的根源。基于这一发现，我们提出动态提示学习（DPL），强制跨注意力映射聚焦于文本提示中的正确名词。通过利用所提出的泄露修复损失更新文本输入中名词的动态标记，我们实现了对特定对象的细粒度图像编辑，同时防止其他图像区域发生非预期改变。我们的方法DPL基于公开的Stable Diffusion模型，在广泛图像上进行了全面评估，在定量指标（CLIP分数、结构差异）和定性评估（用户评价）中均取得卓越结果。我们展示了在词汇替换、提示精炼和注意力重加权任务中改进的提示编辑效果，尤其适用于复杂多目标场景。