Introduced to enhance the efficiency of large language model (LLM) inference, speculative decoding operates by having a smaller model generate a draft. A larger target model then reviews this draft to align with its output, and any acceptance by the target model results in a reduction of the number of the target model runs, ultimately improving efficiency. However, the drafting process in speculative decoding includes slow autoregressive generation and allocates equal time to generating tokens, irrespective of their importance. These inefficiencies collectively contribute to the suboptimal performance of speculative decoding. To further improve LLM inference, we introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. The Vertical Cascade eliminates autoregressive generation from neural models, while the Horizontal Cascade optimizes time allocation in drafting for improved efficiency. Combining both cascades, CS Drafting achieves up to an 81 percent additional speedup over speculative decoding in our experiments, while maintaining the same output distribution as the target model. Our code is publicly available at https://github.com/lfsszd/CS-Drafting.

翻译：为提升大语言模型（LLM）推理效率而引入的推测解码技术，通过让小型模型生成草案来运作。随后更大的目标模型会审查该草案以确保其输出与自身一致，目标模型每接受一个token即可减少一次自身运行次数，从而最终提升效率。然而，推测解码中的草拟过程包含缓慢的自回归生成，并且对token的生成分配相同时间，无论其重要性如何。这些低效因素共同导致推测解码的性能未达最优。为进一步改进LLM推理，我们提出级联推测草拟（CS Drafting），这是一种融合两种级联机制的推测执行算法。垂直级联消除了神经模型的自回归生成过程，而水平级联则优化了草拟阶段的时间分配以提升效率。结合两种级联机制，CS Drafting在实验中相较推测解码实现了最高81%的额外加速，同时保持与目标模型相同的输出分布。我们的代码已开源至https://github.com/lfsszd/CS-Drafting。