Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion Models (MDMs) currently perform best but still underperform AR models in perplexity and lack key inference-time efficiency features, most notably KV caching. We introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms, smoothly interpolating between their perplexities while overcoming their respective limitations. Unlike prior work, which uses transformers with bidirectional attention as MDM denoisers, we exploit the connection between MDMs and Any-Order autoregressive models and adopt causal attention. This design lets us compute the exact likelihood of MDMs for the first time and, crucially, enables us \to introduce KV caching for MDMs while preserving parallel generation for the first time, significantly improving inference efficiency. Combined with an optimized sampling schedule, Eso-LMs achieves a new state of the art on the speed-quality Pareto frontier for unconditional generation. On long contexts, it yields $\mathbf{14 - 65{}\times}$ faster inference than standard MDMs and $\mathbf{3 - 4{}\times}$ faster inference than prior semi-autoregressive approaches. We provide code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/Eso-LMs

翻译：基于扩散的语言模型通过支持并行与可控生成，为自回归模型提供了一种极具吸引力的替代方案。在该模型家族中，掩码扩散模型目前表现最佳，但在困惑度上仍逊于自回归模型，且缺乏关键的推理时效率特性，尤其是KV缓存。我们提出了Eso-LMs这一新模型家族，它融合了自回归与掩码扩散模型范式，能在两者困惑度之间平滑插值，同时克服各自局限。与先前使用具有双向注意力的Transformer作为MDM去噪器的工作不同，我们利用MDM与任意顺序自回归模型之间的关联，采用因果注意力机制。这一设计首次实现了MDM精确似然度的计算，并且关键地首次为MDM引入了KV缓存，同时保留了并行生成能力，显著提升了推理效率。结合优化的采样策略，Eso-LMs在无条件生成任务的速度-质量帕累托边界上达到了新的最优水平。在长上下文场景中，其推理速度比标准MDM快$\mathbf{14 - 65{}\times}$，比先前的半自回归方法快$\mathbf{3 - 4{}\times}$。我们在项目页面提供了代码、模型检查点及视频教程：http://s-sahoo.github.io/Eso-LMs