Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.

翻译：去噪扩散概率模型（DDPMs）（Ho等人，2020）在连续状态空间的图像和波形生成上展示了令人瞩目的成果。本文提出离散去噪扩散概率模型（D3PMs），这是一种适用于离散数据的类扩散生成模型，通过对均匀转移概率的破坏过程进行扩展，推广了Hoogeboom等人（2021）的多项式扩散模型。该扩展包括：模拟连续空间中高斯核的转移矩阵、基于嵌入空间中最近邻的矩阵，以及引入吸收状态的矩阵。第三种矩阵使我们能够建立扩散模型与自回归及基于掩码的生成模型之间的联系。研究表明，转移矩阵的选择是一项重要的设计决策，可在图像和文本领域带来更优的结果。我们还引入了一种新的损失函数，将变分下界与辅助交叉熵损失相结合。在文本方面，该模型类在字符级文本生成上取得了强劲结果，并能扩展至LM1B数据集的大词汇量场景。在图像数据集CIFAR-10上，我们的模型接近连续空间DDPM模型的样本质量，并超越了其对数似然值。