Diffusion models have shown remarkable capabilities in generating high quality and creative images conditioned on text. An interesting application of such models is structure preserving text guided image editing. Existing approaches rely on text conditioned diffusion models such as Stable Diffusion or Imagen and require compute intensive optimization of text embeddings or fine-tuning the model weights for text guided image editing. We explore text guided image editing with a Hybrid Diffusion Model (HDM) architecture similar to DALLE-2. Our architecture consists of a diffusion prior model that generates CLIP image embedding conditioned on a text prompt and a custom Latent Diffusion Model trained to generate images conditioned on CLIP image embedding. We discover that the diffusion prior model can be used to perform text guided conceptual edits on the CLIP image embedding space without any finetuning or optimization. We combine this with structure preserving edits on the image decoder using existing approaches such as reverse DDIM to perform text guided image editing. Our approach, PRedItOR does not require additional inputs, fine-tuning, optimization or objectives and shows on par or better results than baselines qualitatively and quantitatively. We provide further analysis and understanding of the diffusion prior model and believe this opens up new possibilities in diffusion models research.

翻译：扩散模型在基于文本生成高质量且富有创意的图像方面展现出卓越能力。此类模型的一个有趣应用是保持结构的文本引导图像编辑。现有方法依赖文本条件扩散模型（如Stable Diffusion或Imagen），需要对文本嵌入进行计算密集型优化或微调模型权重以实现文本引导图像编辑。我们探索了使用类似DALLE-2的混合扩散模型架构进行文本引导图像编辑。该架构包含一个扩散先验模型（生成基于文本提示的CLIP图像嵌入）和一个自定义的潜扩散模型（训练生成基于CLIP图像嵌入的图像）。我们发现，扩散先验模型可在无需任何微调或优化的条件下对CLIP图像嵌入空间执行文本引导的概念编辑。我们将其与现有方法（如反向DDIM）对图像解码器进行结构保持编辑相结合，以实现文本引导图像编辑。我们的方法PRedItOR无需额外输入、微调、优化或目标函数，在定性与定量评估中均达到与基线方法相当或更优的效果。我们进一步分析了扩散先验模型，并相信这为扩散模型研究开辟了新可能。