The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them quickly and cheaply. This paper introduces SpecInfer, an LLM serving system that accelerates generative LLM inference with speculative inference and token tree verification. A key insight behind SpecInfer is to combine various collectively boost-tuned small language models to jointly predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified by the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality.

翻译：生成式大语言模型高昂的计算和内存需求使其难以实现快速且低成本的服务。本文提出SpecInfer，一种通过推测推理与令牌树验证加速生成式大语言模型推理的服务系统。其核心思想在于使用多种联合微调的小型语言模型共同预测大语言模型的输出，并将预测结果组织成令牌树结构，每个节点代表一个候选令牌序列。通过基于树结构的新型并行解码机制，大语言模型可同时验证令牌树中所有候选序列的正确性。SpecInfer将大语言模型作为令牌树验证器而非增量解码器，在可证明保持模型质量的前提下，显著降低了生成式大语言模型服务的端到端延迟和计算需求。