Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of MineDraft in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

翻译：推测解码通过使用较小的草稿模型生成候选令牌，再由较大的目标模型进行验证，从而加速大语言模型推理。然而，标准推测解码的性能通常受限于草稿与验证阶段严格的串行执行。为此，本文提出MineDraft——一种批量并行推测解码框架，旨在通过重叠验证与草稿生成过程来有效隐藏草稿延迟。理论分析表明，批量并行推测解码的效率显著优于标准推测解码。MineDraft通过新颖的批量并行设计实现该框架，维持两个请求批次并使得一个批次的草稿生成与另一批次的验证重叠。实验结果显示，与标准推测解码相比，MineDraft在吞吐量（提升高达75%）和端到端延迟（降低高达39%）方面均有显著改进。此外，我们已将MineDraft实现为vLLM的插件，展示了其在生产级推理系统中的实用性。