Diffusion models have emerged as a powerful paradigm for modern generative modeling, demonstrating strong potential for large language models (LLMs). Unlike conventional autoregressive (AR) models that generate tokens sequentially, diffusion models allow for parallel sampling, offering a promising path to accelerate generation and eliminate the left-to-right generation constraints. Despite their empirical success, theoretical understandings of diffusion language models remain underdeveloped. In this work, we develop convergence guarantees for diffusion language models from an information-theoretic perspective. Our analysis demonstrates that the sampling error, measured by the Kullback-Leibler (KL) divergence, decays inversely with the number of iterations $T$ and scales linearly with the mutual information between tokens in the target text sequence. Crucially, our theory covers the regime $T<L$, where $L$ is the text sequence length. This justifies that high-quality samples can be generated with fewer iterations than $L$, thereby breaking the fundamental sampling bottleneck of $L$ steps required by AR models. We further establish matching upper and lower bounds, up to some constant factor, that shows the tightness of our convergence analysis. These results offer novel theoretical insights into the practical effectiveness of diffusion language models.

翻译：扩散模型已成为现代生成建模的强大范式，在大语言模型（LLMs）领域展现出巨大潜力。与传统的自回归（AR）模型逐词元顺序生成不同，扩散模型支持并行采样，为加速生成和消除从左到右的生成约束提供了可行路径。尽管扩散模型在实证上取得了成功，但其理论理解仍不充分。本文从信息论角度为扩散语言模型建立了收敛性保证。我们的分析表明，以Kullback-Leibler（KL）散度衡量的采样误差随迭代次数$T$呈反比衰减，并与目标文本序列中词元间的互信息呈线性关系。关键的是，我们的理论覆盖了$T<L$的区间，其中$L$为文本序列长度。这证明了高质量样本可以在少于$L$次迭代中生成，从而突破了AR模型所需$L$步的基本采样瓶颈。我们进一步建立了匹配的上下界（至常数因子），证明了收敛分析的紧致性。这些结果为扩散语言模型的实际有效性提供了新的理论见解。