Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\times$ speedup over vanilla LLaDA/Dream and 5$\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM.

翻译：扩散大语言模型（dLLMs）提供了超越自回归（AR）大语言模型的能力，例如并行解码和随机顺序生成。然而，在实践中实现这些优势并非易事，因为dLLM本质上面临准确性与并行性之间的权衡。尽管关注度日益增加，但现有方法通常只关注问题的一个方面，即要么针对效率，要么针对性能。为了解决这一局限性，我们提出了d3LLM（伪蒸馏扩散大语言模型），在准确性与并行性之间取得了平衡：（i）在训练阶段，我们引入伪轨迹蒸馏，以教导模型哪些词元可以在早期步骤中被自信地解码，从而提高并行性；（ii）在推理阶段，我们采用基于熵的多块解码与KV缓存刷新机制，在保持准确性的同时实现高并行性。为了更好地评估dLLM，我们还引入了AUP（并行性下的准确性），这是一个同时衡量准确性与并行性的新指标。实验表明，我们的d3LLM相比原始LLaDA/Dream实现了高达10倍的加速，相比AR模型实现了5倍的加速，且准确性下降不大。我们的代码可在 https://github.com/hao-ai-lab/d3LLM 获取。