We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.

翻译：我们提出一种推测采样算法，通过使每次Transformer调用生成多个token来加速解码过程。该算法基于以下观察：由速度更快但能力较弱的草稿模型生成的短文本续写，其并行评分延迟与直接从更大目标模型采样单个token的延迟相当。我们结合一种新颖的改进型拒绝采样方案，该方案在硬件数值精度内保留了目标模型的分布特性。我们以拥有700亿参数的Chinchilla语言模型为基准测试推测采样，在分布式环境下实现了2-2.5倍解码加速，且未妥协样本质量或对模型本身进行任何修改。