Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their widespread application is hindered by the resource-intensive decoding process. To address this challenge, current approaches have incorporated additional decoding heads to enable parallel prediction of multiple subsequent tokens, thereby achieving inference acceleration. Nevertheless, the accuracy of these decoding heads falls short of the auto-regressive decoding approach. In light of these limitations, we propose Chimera, a novel framework specifically designed for speculative sampling. Within this framework, we introduce a lightweight draft model that effectively utilizes previously generated tokens to predict subsequent words. To ensure both accuracy and efficiency, we present two strategies within the lightweight draft model. Firstly, we focus on capturing short-range dependencies at the bottom layer. Secondly, we leverage the readily available representations from the original LLM.Through empirical evaluation on the Vicuna and LlaMA-2 series, Chimera demonstrates impressive results, achieving an average latency speedup ratio of 2.7x compared to the vanilla auto-regressive decoding approach. This highlights the potential of our proposed framework in significantly improving the efficiency of large language models during the decoding process.

翻译：大语言模型（LLM）在各类任务中展现出卓越能力，但其资源密集型的解码过程限制了广泛应用。当前方法通过引入额外解码头并行预测后续多个令牌来实现推理加速，然而这些解码头的准确性仍不及自回归解码方式。针对这些局限，我们提出Chimera——一种专为推测采样设计的新型框架。该框架引入轻量级草稿模型，通过有效利用已生成令牌来预测后续词语。为确保准确性与效率，我们在轻量级草稿模型中提出两种策略：首先在底层捕获短程依赖关系，其次充分利用原始LLM中现成的表征。基于Vicuna和LlaMA-2系列的实证评估表明，与标准自回归解码方法相比，Chimera实现了平均2.7倍的延迟加速比，充分彰显了本框架在显著提升大语言模型解码效率方面的潜力。