Deployment of autoregressive large language models (LLMs) is costly, and as these models increase in size, the associated costs will become even more considerable. Consequently, different methods have been proposed to accelerate the token generation process and reduce costs. Speculative decoding (SD) is among the most promising approaches to speed up the LLM decoding process by verifying multiple tokens in parallel and using an auxiliary smaller draft model to generate the possible tokens. In SD, usually, one draft model is used to serve a specific target model; however, in practice, LLMs are diverse, and we might need to deal with many target models or more than one target model simultaneously. In this scenario, it is not clear which draft model should be used for which target model, and searching among different draft models or training customized draft models can further increase deployment costs. In this paper, we first introduce a novel multi-target scenario for the deployment of draft models for faster inference. Then, we present a novel, more efficient sorted speculative decoding mechanism that outperforms regular baselines in multi-target settings. We evaluated our method on Spec-Bench in different settings, including base models such as Vicuna 7B, 13B, and LLama Chat 70B. Our results suggest that our draft models perform better than baselines for multiple target models at the same time.

翻译：自回归大语言模型（LLM）的部署成本高昂，且随着模型规模扩大，相关成本将进一步攀升。因此，学界提出了多种加速令牌生成过程以降低开销的方法。推测解码（SD）通过并行验证多个令牌并利用辅助的小型草稿模型生成候选令牌，成为加速LLM解码过程最具前景的技术之一。传统SD通常采用单一草稿模型服务特定目标模型；然而实际应用中LLM具有多样性，我们可能需要同时处理多个目标模型或多样化的目标模型集合。在此场景下，如何为不同目标模型匹配合适的草稿模型成为难题，而遍历不同草稿模型或训练定制化草稿模型将进一步增加部署成本。本文首先针对加速推理的草稿模型部署提出了创新的多目标场景框架，进而提出一种新颖高效的排序推测解码机制，其在多目标设定下优于常规基线方法。我们在Spec-Bench上使用不同基础模型（包括Vicuna 7B、13B和LLama Chat 70B）进行了多场景评估。实验结果表明，我们的草稿模型在同时服务多个目标模型时均优于基线方法。