Parallel decoding methods such as Jacobi decoding show promise for more efficient LLM inference as it breaks the sequential nature of the LLM decoding process and transforms it into parallelizable computation. However, in practice, it achieves little speedup compared to traditional autoregressive (AR) decoding, primarily because Jacobi decoding seldom accurately predicts more than one token in a single fixed-point iteration step. To address this, we develop a new approach aimed at realizing fast convergence from any state to the fixed point on a Jacobi trajectory. This is accomplished by refining the target LLM to consistently predict the fixed point given any state as input. Extensive experiments demonstrate the effectiveness of our method, showing 2.4$\times$ to 3.4$\times$ improvements in generation speed while preserving generation quality across both domain-specific and open-domain benchmarks.

翻译：摘要：诸如雅可比解码（Jacobi decoding）等并行解码方法有望提高大型语言模型（LLM）的推理效率，因为它打破了LLM解码过程的顺序特性，将其转化为可并行计算。然而在实践中，与传统的自回归（AR）解码相比，该方法仅能带来微小的加速，主要原因在于雅可比解码在单次不动点迭代步骤中很少能准确预测多个令牌。为解决这一问题，我们开发了一种新方法，旨在实现从雅可比轨迹上任意状态到不动点的快速收敛。这是通过优化目标LLM，使其在给定任意状态作为输入时能够一致地预测不动点来实现的。大量实验证明了我们方法的有效性，在领域特定和开放领域基准测试中，生成速度提升了2.4倍至3.4倍，同时保持了生成质量。