Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel. However, this method presents a critical trade-off: it improves throughput in low-load, memory-bound systems but degrades performance in high-load, compute-bound environments due to verification overhead. Existing speculative decoding methods use fixed lengths and cannot adapt to workload changes or decide when to stop speculation. The cost of restarting speculative inference also remains unquantified. Under high load, the benefit of speculation diminishes, while retaining the draft model reduces KV cache capacity, limiting batch size and degrading throughput. To overcome this, we propose Nightjar, a resource-aware adaptive speculative framework. It first adjusts to the request load by dynamically selecting the optimal speculative length for different batch sizes. Crucially, Nightjar proactively disables speculative decoding when the MAB planner determines that speculation is no longer beneficial, and during the disabled phase, offloads the draft model to the CPU only under GPU memory pressure. This reclaims memory for the KV cache, thereby facilitating larger batch sizes and maximizing overall system throughput. Experiments show that Nightjar achieves up to 14.76% higher throughput than standard speculative decoding and up to 20.18% lower latency in the main benchmark suite under dynamic request arrival rates for real-time LLM serving scenarios.

翻译：推测解码（SD）通过并行验证草稿令牌加速大语言模型推理。然而该方法存在关键权衡：在低负载、内存受限系统中能提升吞吐量，但在高负载、计算受限环境中因验证开销导致性能下降。现有推测解码方法使用固定长度，无法适应负载变化或决定何时停止推测，推测推理重启代价仍未被量化。高负载下推测收益递减，而保留草稿模型会减少KV缓存容量，限制批处理尺寸并降低吞吐量。为此，我们提出资源感知的自适应推测框架Nightjar。该框架首先通过为不同批处理尺寸动态选择最优推测长度来适应请求负载。关键在于，当多臂老虎机（MAB）规划器判定推测不再有益时，Nightjar主动禁用推测解码，并在禁用阶段仅在GPU内存压力下将草稿模型卸载至CPU。此举为KV缓存回收内存，从而支持更大批处理尺寸并最大化系统总吞吐量。实验表明，在实时LLM服务场景的动态请求到达率下，Nightjar在主要基准测试套件中相比标准推测解码实现最高14.76%的吞吐量提升，时延降低20.18%。