Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressively, requiring a forward pass for each generated token, and thus reading the full set of parameters from memory. This memory access forms the primary bottleneck for generation and it worsens as the model size increases. Moreover, executing a forward pass for multiple tokens in parallel often takes nearly the same time as it does for just one token. These two observations lead to the development of speculative sampling, where a second smaller model is used to draft a few tokens, that are then validated or rejected using a single forward pass of the large model. Unfortunately, this method requires two models that share the same tokenizer and thus limits its adoption. As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model. Our approach only requires an additional input token that marks the words that will be generated simultaneously. We show promising performance (up to $30\%$ speed-up) while requiring only as few as $O(d_{emb})$ additional parameters.

翻译：将语言模型的规模扩展至数百亿参数，已在多种任务上取得了显著性能。在生成过程中，这些模型采用自回归方式运行，每生成一个token就需要一次前向传播，从而需要从内存中读取全部参数集。这种内存访问成为生成的主要瓶颈，并且随着模型规模增大而加剧。此外，并行执行多个token的前向传播所需时间几乎与单个token相同。这两个观察结果催生了推测采样方法——该方法利用一个较小的辅助模型草拟若干token，然后通过大模型的单次前向传播对这些token进行验证或拒绝。遗憾的是，该方法要求两个模型共享相同的分词器，从而限制了其应用范围。作为替代方案，我们提出利用并行解码技术从单个模型中生成多个草拟token，既无需计算开销，也不需要第二个模型。我们的方法仅需额外添加一个输入标记，用于标识将同时生成的词语。实验表明，在仅需$O(d_{emb})$量级额外参数的情况下，该方法取得了最高$30\%$的加速效果。