This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Our key insight is -- learning diffusion models for marginal, conditional, and joint distributions can be unified as predicting the noise in the perturbed data, where the perturbation levels (i.e. timesteps) can be different for different modalities. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model -- perturbs data in all modalities instead of a single modality, inputs individual timesteps in different modalities, and predicts the noise of all modalities instead of a single modality. UniDiffuser is parameterized by a transformer for diffusion models to handle input types of different modalities. Implemented on large-scale paired image-text data, UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead. In particular, UniDiffuser is able to produce perceptually realistic samples in all tasks and its quantitative results (e.g., the FID and CLIP score) are not only superior to existing general-purpose models but also comparable to the bespoken models (e.g., Stable Diffusion and DALL-E 2) in representative tasks (e.g., text-to-image generation).

翻译：本文提出统一扩散框架（称UniDiffuser），旨在通过单一模型拟合多模态数据涉及的所有分布。核心洞见在于：边际分布、条件分布与联合分布的扩散模型学习可统一为预测扰动数据中的噪声，不同模态的扰动水平（即时间步）可独立设置。基于该统一视角，UniDiffuser对原始扩散模型进行最小化修改即可同步学习所有分布——扰动所有模态而非单一模态数据，为不同模态输入独立时间步，并预测所有模态而非单一模态的噪声。UniDiffuser采用适用于扩散模型的Transformer架构参数化，以处理多模态数据的输入类型。基于大规模图文配对数据实现的UniDiffuser，可通过设置适当时间步执行图像生成、文本生成、文生图、图生文及图文对生成任务，且无需额外开销。特别地，UniDiffuser在所有任务中均能生成感知逼真的样本，其定量指标（如FID与CLIP评分）不仅优于现有通用模型，在代表性任务（如文生图生成）上更可与专用模型（如Stable Diffusion及DALL-E 2）相媲美。