With the increasingly giant scales of (causal) large language models (LLMs), the inference efficiency comes as one of the core concerns along the improved performance. In contrast to the memory footprint, the latency bottleneck seems to be of greater importance as there can be billions of requests to a LLM (e.g., GPT-4) per day. The bottleneck is mainly due to the autoregressive innateness of LLMs, where tokens can only be generated sequentially during decoding. To alleviate the bottleneck, the idea of speculative execution, which originates from the field of computer architecture, is introduced to LLM decoding in a \textit{draft-then-verify} style. Under this regime, a sequence of tokens will be drafted in a fast pace by utilizing some heuristics, and then the tokens shall be verified in parallel by the LLM. As the costly sequential inference is parallelized, LLM decoding speed can be significantly boosted. Driven by the success of LLMs in recent couple of years, a growing literature in this direction has emerged. Yet, there lacks a position survey to summarize the current landscape and draw a roadmap for future development of this promising area. To meet this demand, we present the very first survey paper that reviews and unifies literature of speculative execution in LLMs (e.g., blockwise parallel decoding, speculative decoding, etc.) in a comprehensive framework and a systematic taxonomy. Based on the taxonomy, we present a critical review and comparative analysis of the current arts. Finally we highlight various key challenges and future directions to further develop the area.

翻译：随着（因果）大语言模型规模的日益庞大，推理效率成为与性能提升并列的核心关注点之一。相较于内存占用，延迟瓶颈显得更为重要，因为一个大型语言模型（例如GPT-4）每天可能面临数十亿次请求。该瓶颈主要源于大语言模型的自回归本质，即在解码过程中令牌只能依次顺序生成。为缓解这一瓶颈，源于计算机体系结构领域的“推测执行”思想，以“草拟-验证”的方式被引入到大语言模型解码中。在该范式下，通过利用某些启发式策略快速草拟一个令牌序列，然后由大语言模型并行验证这些令牌。由于昂贵的顺序推理被并行化，大语言模型的解码速度得以显著提升。受近年大语言模型成功应用的驱动，该方向涌现出日益增多的文献。然而，目前尚缺乏一份定位性综述来总结当前研究格局，并为这一前景广阔的领域绘制未来发展路线图。为满足这一需求，我们提出了首篇综述论文，该论文在综合框架和系统化分类体系下，回顾并统一了大语言模型中推测执行（例如块级并行解码、推测解码等）的相关文献。基于该分类体系，我们对现有技术进行了评述性回顾与比较分析。最后，我们指出了进一步发展该领域所需应对的关键挑战与未来方向。