The quality of the prompts provided to text-to-image diffusion models determines how faithful the generated content is to the user's intent, often requiring `prompt engineering'. To harness visual concepts from target images without prompt engineering, current approaches largely rely on embedding inversion by optimizing and then mapping them to pseudo-tokens. However, working with such high-dimensional vector representations is challenging because they lack semantics and interpretability, and only allow simple vector operations when using them. Instead, this work focuses on inverting the diffusion model to obtain interpretable language prompts directly. The challenge of doing this lies in the fact that the resulting optimization problem is fundamentally discrete and the space of prompts is exponentially large; this makes using standard optimization techniques, such as stochastic gradient descent, difficult. To this end, we utilize a delayed projection scheme to optimize for prompts representative of the vocabulary space in the model. Further, we leverage the findings that different timesteps of the diffusion process cater to different levels of detail in an image. The later, noisy, timesteps of the forward diffusion process correspond to the semantic information, and therefore, prompt inversion in this range provides tokens representative of the image semantics. We show that our approach can identify semantically interpretable and meaningful prompts for a target image which can be used to synthesize diverse images with similar content. We further illustrate the application of the optimized prompts in evolutionary image generation and concept removal.

翻译：向文本到图像扩散模型提供的提示质量决定了生成内容与用户意图的契合程度，通常需要“提示工程”。为在不依赖提示工程的情况下从目标图像中提取视觉概念，现有方法主要依赖嵌入反演——通过优化嵌入向量并将其映射为伪标记。然而，这类高维向量表示因缺乏语义性和可解释性，且仅支持简单向量运算而面临挑战。为此，本研究聚焦于直接反演扩散模型以获得可解释的语言提示。其难点在于：由此产生的优化问题本质上是离散的，提示空间呈指数级增长，这使得随机梯度下降等标准优化技术难以应用。针对此问题，我们采用延迟投影方案来优化代表模型词汇空间的提示。此外，我们利用扩散过程不同时间步对应图像不同细节层次的特性：前向扩散过程中的后期含噪声时间步对应语义信息，因此在此区间进行提示反演可获得代表图像语义的标记。实验表明，该方法能够为目标图像识别出语义可解释且有意义的提示，从而合成内容相似的多样化图像。我们进一步展示了优化提示在进化图像生成与概念移除中的应用。