Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\times$ speedup over vanilla LLaDA/Dream and 5$\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM.

翻译：扩散大语言模型（dLLMs）提供了超越自回归（AR）大语言模型的能力，例如并行解码和随机顺序生成。然而，在实践中实现这些优势并非易事，因为dLLMs本质上面临着准确性与并行性的权衡。尽管关注度日益增加，现有方法通常只关注问题的一个方面，即要么针对效率，要么针对性能。为了克服这一局限，我们提出了d3LLM（伪蒸馏扩散大语言模型），在准确性与并行性之间取得了平衡：（i）在训练阶段，我们引入伪轨迹蒸馏来教导模型哪些令牌可以在早期步骤被自信地解码，从而提升并行性；（ii）在推理阶段，我们采用基于熵的多块解码配合KV缓存刷新机制，以在保持准确性的同时实现高并行性。为了更好地评估dLLMs，我们还引入了AUP（并行性下的准确性），这是一个同时衡量准确性与并行性的新指标。实验表明，我们的d3LLM相比原始LLaDA/Dream实现了高达10倍的加速，相比AR模型实现了5倍的加速，且准确性下降不大。我们的代码可在 https://github.com/hao-ai-lab/d3LLM 获取。