Language models with recurrent depth, also referred to as universal or looped when considering transformers, are defined by the capacity to increase their computation through the repetition of layers. Recent efforts in pretraining have demonstrated that these architectures can scale to modern language modeling tasks while exhibiting advantages in reasoning tasks. In this work, we examine the relationship between recurrent-depth models and diffusion language models. Building on their similarities, we develop a new diffusion forcing sampler for these models to accelerate generation. The sampler advances by decoding new tokens at every forward pass of the model, while the latent states of these tokens can be further refined in parallel through recurrence. Theoretically, generation with our sampler is strictly more expressive than the baseline autoregressive generation using the same time budget on modern hardware. Moreover, this sampler, based on principles from diffusion literature, can be directly applied to existing 3.5B recurrent-depth transformers without any tuning, leading to up to a 5x speedup. Consequently, our findings not only provide an efficient mechanism for parallelizing the extra computation in recurrent-depth models at inference, but also suggest that such models can be naturally viewed as strong continuous, though causal, diffusion language models.

翻译：循环深度语言模型（在考虑Transformer架构时亦称为通用或循环模型）通过层重复实现计算能力的增强。近期预训练研究表明，此类架构能够扩展至现代语言建模任务，并在推理任务中展现出优势。本文探讨循环深度模型与扩散语言模型之间的理论关联。基于二者的相似性，我们为循环深度模型开发了一种新型扩散强制采样器以加速生成过程。该采样器通过在模型的每次前向传播中解码新词元实现推进，同时这些词元的潜在状态可通过循环机制进行并行精细化处理。从理论上证明，在相同时间预算下，采用本采样器的生成方式比基于现代硬件的标准自回归生成具有更严格的表达能力。此外，这种基于扩散理论原理的采样器无需任何调优即可直接应用于现有的35亿参数循环深度Transformer模型，实现高达5倍的加速效果。因此，我们的发现不仅为推理过程中并行化循环深度模型的额外计算提供了高效机制，同时表明此类模型可被自然地视为强大的连续（尽管具有因果性）扩散语言模型。