We propose an acceleration scheme for large language models (LLMs) through Speculative Decoding with Semantic Adaptive Tokens (SDSAT). The primary objective of this design is to enhance the LLM model's ability to generate draft tokens more accurately without compromising the model's accuracy. The core strategies involve: 1) Fine-tune the model by incorporating semantic adaptive tokens that possess flexible decoding capabilities without changing its structure, allowing them to generate high-quality draft tokens. 2) By employing a training method that does not affect the standard tokens, the model can acquire parallel decoding abilities atop its original framework with minimal training overhead. 3) We have designed the "two-step-draft-then-verify" generation strategies using both greedy search and nucleus sampling. Experiments conducted on the CodeLlama-13B and 7B models have yielded speed increases of over 3.5X and 3.0X, respectively. Please refer to https://github.com/hasuoshenyun/SDSAT.

翻译：我们提出了一种通过语义自适应令牌的推测解码（SDSAT）加速大语言模型（LLMs）推理的方案。该设计的核心目标是在不损害模型精度的前提下，提升LLM模型生成草稿令牌的准确性。核心策略包括：1）在不改变模型结构的情况下，通过引入具有灵活解码能力的语义自适应令牌对模型进行微调，使其能够生成高质量的草稿令牌。2）采用不影响标准令牌的训练方法，使模型在原始框架之上以极小的训练开销获得并行解码能力。3）我们设计了基于贪婪搜索和核采样的"两步草稿-验证"生成策略。在CodeLlama-13B和7B模型上进行的实验表明，速度分别提升了3.5倍和3.0倍以上。详见https://github.com/hasuoshenyun/SDSAT。