We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the same task on a new query image following the text guidance. To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly over six different tasks using these prompts. The resulting Prompt Diffusion model is the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation on the trained tasks and generalizes effectively to new, unseen vision tasks with their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Prompt-Diffusion.

翻译：我们提出Prompt Diffusion，一种在基于扩散的生成模型中实现上下文学习的框架。给定任务特定的示例图像对（例如深度到图像和涂鸦到图像）以及文本引导，我们的模型自动理解底层任务，并根据文本引导对新的查询图像执行相同任务。为实现这一点，我们提出了一种能够建模广泛视觉-语言任务的视觉-语言提示，以及一个将其作为输入的扩散模型。该扩散模型使用这些提示在六个不同任务上联合训练。最终得到的Prompt Diffusion模型是首个具备上下文学习能力的基于扩散的视觉-语言基础模型。它在训练任务上展现出高质量的上下文生成效果，并有效泛化到具有相应提示的新颖未见视觉任务。我们的模型还展示了令人信服的文本引导图像编辑结果。我们的框架旨在推动计算机视觉中上下文学习的研究。我们在https://github.com/Zhendong-Wang/Prompt-Diffusion分享代码和预训练模型。