The goal of $L$-step speculative decoding is to accelerate autoregressive decoding of a target model by using a cheaper draft model to generate a candidate path of $L$ tokens. Based on a verification algorithm involving target and draft model probabilities, a prefix of the candidate sequence is accepted, and an additional correction token is sampled from a residual distribution to ensure that the final output adheres to the target distribution. While standard speculative decoding uses a verification algorithm which is independent at each token on the path, a recent extension called block verification uses a joint condition involving all sampled on-path probabilities. Block verification (BV) was shown to be optimal over all verification algorithms which use only on-path probabilities, improving on standard speculative decoding. In this work, we first show that block verification is optimal even over verification algorithms that use off-path probabilities, by constructing an information-agnostic linear program (LP). Further, we can extend our LP to the setting where the draft model samples multiple candidate paths, and use it to construct a natural class of multi-path block verification generalizations. While computing the optimal algorithm in this class is not tractable, by considering a stricter class of greedy algorithms, we can formulate an efficient method called greedy multi-path block verification (GBV). Empirically, GBV can improve block efficiency by over 30% and reduce decoding walltimes by over 15% relative to BV. On Llama-3 70B, GBV can improve the end-to-end decoding throughput over SOTA multi-path verification methods by more than 15%.

翻译：$L$步推测解码的目标是通过使用更廉价的草稿模型生成包含$L$个标记的候选路径，从而加速目标模型的自回归解码。基于涉及目标模型和草稿模型概率的验证算法，候选序列的一个前缀被接受，并从残差分布中采样一个额外的修正标记，以确保最终输出符合目标分布。虽然标准推测解码使用的验证算法在路径的每个标记处独立，但最近提出的块验证扩展采用了一个涉及所有路径上采样概率的联合条件。块验证（BV）被证明在所有仅使用路径上概率的验证算法中是最优的，优于标准推测解码。在本工作中，我们首先通过构建一个信息无关的线性规划（LP），证明块验证即使在可以使用路径外概率的验证算法中也是最优的。进一步，我们可以将线性规划扩展到草稿模型采样多条候选路径的设置，并用它构建一类自然的多路径块验证推广方法。虽然计算此类中的最优算法是棘手的，但通过考虑一类更严格的贪婪算法，我们可以提出一种称为贪婪多路径块验证（GBV）的高效方法。实验表明，相对于BV，GBV可以将块效率提升超过30%，并将解码时间减少超过15%。在Llama-3 70B上，GBV可以将端到端解码吞吐量相较于最先进的多路径验证方法提升超过15%。