Despite a growing interest in diffusion-based language models, existing work has not shown that these models can attain nontrivial likelihoods on standard language modeling benchmarks. In this work, we take the first steps towards closing the likelihood gap between autoregressive and diffusion-based language models, with the goal of building and releasing a diffusion model which outperforms a small but widely-known autoregressive model. We pursue this goal through algorithmic improvements, scaling laws, and increased compute. On the algorithmic front, we introduce several methodological improvements for the maximum-likelihood training of diffusion language models. We then study scaling laws for our diffusion models and find compute-optimal training regimes which differ substantially from autoregressive models. Using our methods and scaling analysis, we train and release Plaid 1B, a large diffusion language model which outperforms GPT-2 124M in likelihood on benchmark datasets and generates fluent samples in unconditional and zero-shot control settings.

翻译：尽管基于扩散的语言模型引起了越来越多的关注，但现有工作尚未证明这些模型能在标准语言建模基准上实现显著的似然值。在本工作中，我们迈出了缩小自回归与扩散语言模型之间似然差距的第一步，目标在于构建并发布一个在性能上超越小型但广泛知名的自回归模型的扩散模型。我们通过算法改进、缩放定律以及增加计算量来实现这一目标。在算法层面，我们引入了若干针对扩散语言模型最大似然训练的方法论改进。随后，我们研究了扩散模型的缩放定律，并发现了与自回归模型截然不同的计算最优训练模式。利用我们的方法与缩放分析，我们训练并发布了Plaid 1B——一个大型扩散语言模型，它在基准数据集上的似然表现优于GPT-2 124M，并在无条件与零样本控制场景中生成流畅的样本。