The modern autoregressive Large Language Models (LLMs) have achieved outstanding performance on NLP benchmarks, and they are deployed in the real world. However, they still suffer from limitations of the autoregressive training paradigm. For example, autoregressive token generation is notably slow and can be prone to \textit{exposure bias}. The diffusion-based language models were proposed as an alternative to autoregressive generation to address some of these limitations. We evaluate the recently proposed Score Entropy Discrete Diffusion (SEDD) approach and show it is a promising alternative to autoregressive generation but it has some short-comings too. We empirically demonstrate the advantages and challenges of SEDD, and observe that SEDD generally matches autoregressive models in perplexity and on benchmarks such as HellaSwag, Arc or WinoGrande. Additionally, we show that in terms of inference latency, SEDD can be up to 4.5$\times$ more efficient than GPT-2. While SEDD allows conditioning on tokens at abitrary positions, SEDD appears slightly weaker than GPT-2 for conditional generation given short prompts. Finally, we reproduced the main results from the original SEDD paper.

翻译：现代自回归大型语言模型（LLMs）在自然语言处理基准测试中取得了卓越的性能，并已在现实世界中部署应用。然而，它们仍受限于自回归训练范式的固有缺陷。例如，自回归的令牌生成速度显著较慢，且容易受到**暴露偏差**的影响。基于扩散的语言模型被提出作为自回归生成的一种替代方案，以解决部分此类问题。我们评估了近期提出的分数熵离散扩散（SEDD）方法，并证明其是一种有前景的自回归生成替代方案，但也存在一些不足之处。我们通过实证研究展示了SEDD的优势与挑战，并观察到SEDD在困惑度以及HellaSwag、Arc或WinoGrande等基准测试上通常与自回归模型表现相当。此外，在推理延迟方面，SEDD的效率可比GPT-2高出4.5倍。尽管SEDD允许在任意位置对令牌进行条件化生成，但在给定短提示的条件下生成任务中，SEDD的表现略弱于GPT-2。最后，我们复现了原始SEDD论文中的主要结果。