Cloud-based Large Language Model (LLM) services often face challenges in achieving low inference latency and meeting Service Level Objectives (SLOs) under dynamic request patterns. Speculative decoding, which exploits lightweight models for drafting and LLMs for verification, has emerged as a compelling technique to accelerate LLM inference. However, existing speculative decoding solutions often fail to adapt to fluctuating workloads and dynamic system environments, resulting in impaired performance and SLO violations. In this paper, we introduce AdaSpec, an efficient LLM inference system that dynamically adjusts speculative strategies according to real-time request loads and system configurations. AdaSpec proposes a theoretical model to analyze and predict the efficiency of speculative strategies across diverse scenarios. Additionally, it implements intelligent drafting and verification algorithms to maximize performance while ensuring high SLO attainment. Experimental results on real-world LLM service traces demonstrate that AdaSpec consistently meets SLOs and achieves substantial performance improvements, delivering up to 66% speedup compared to state-of-the-art speculative inference systems. The source code is publicly available at https://github.com/cerebellumking/AdaSpec

翻译：基于云的大语言模型（LLM）服务在动态请求模式下，常面临实现低推理延迟和满足服务等级目标（SLO）的挑战。推测解码利用轻量级模型进行草稿生成，并依赖LLM进行验证，已成为加速LLM推理的一项引人注目的技术。然而，现有的推测解码方案通常无法适应波动的工作负载和动态的系统环境，导致性能受损和SLO违规。本文提出AdaSpec，一种高效的LLM推理系统，能够根据实时请求负载和系统配置动态调整推测策略。AdaSpec提出了一个理论模型，用于分析和预测不同场景下推测策略的效率。此外，它实现了智能的草稿生成与验证算法，在确保高SLO达成率的同时最大化性能。在真实世界LLM服务轨迹上的实验结果表明，AdaSpec能够持续满足SLO，并实现显著的性能提升，相比最先进的推测推理系统，最高可带来66%的加速。源代码已公开于 https://github.com/cerebellumking/AdaSpec