Scaling test-time compute has driven the recent advances in the reasoning capabilities of large language models (LLMs), typically by allocating additional computation for more thorough exploration. However, increased compute often comes at the expense of higher user-facing latency, directly impacting user experience. Current test-time scaling methods primarily optimize for accuracy based on total compute resources (FLOPS), often overlooking latency constraints. To address this gap, we propose $\texttt{SPECS}$, a latency-aware test-time scaling method inspired by speculative decoding. $\texttt{SPECS}$~uses a smaller, faster model to generate candidate sequences efficiently, and evaluates these candidates using signals from both a larger target model and a dedicated reward model. We introduce new integration strategies, including reward-guided soft verification and a reward-based deferral mechanism. Empirical results on MATH500, AMC23 and OlympiadBench datasets show that $\texttt{SPECS}$~matches or surpasses beam search accuracy while reducing latency by up to $\sim$19.1\%. Our theoretical analysis shows that our algorithm converges to the solution of a KL-regularized reinforcement learning objective with increasing beam width.

翻译：扩展测试时计算量是推动大语言模型（LLM）推理能力近期取得进展的关键，通常通过分配额外计算资源以实现更彻底的探索。然而，计算量的增加往往以更高的用户端延迟为代价，直接影响用户体验。当前的测试时扩展方法主要基于总计算资源（FLOPS）优化准确率，常常忽视延迟约束。为弥补这一不足，我们提出$\texttt{SPECS}$，一种受推测解码启发的、具有延迟感知能力的测试时扩展方法。$\texttt{SPECS}$利用一个更小、更快的模型高效生成候选序列，并通过来自更大目标模型和专用奖励模型的信号来评估这些候选序列。我们引入了新的集成策略，包括奖励引导的软验证和基于奖励的延迟决策机制。在MATH500、AMC23和OlympiadBench数据集上的实验结果表明，$\texttt{SPECS}$在降低高达$\sim$19.1\%延迟的同时，达到或超越了束搜索的准确率。我们的理论分析表明，随着束宽增加，我们的算法收敛于一个KL正则化强化学习目标的解。