Many applications today provide users with multiple auto-complete drafts as they type, including GitHub's code completion, Gmail's smart compose, and Apple's messaging auto-suggestions. Under the hood, language models support this by running an autoregressive inference pass to provide a draft. Consequently, providing $k$ drafts to the user requires running an expensive language model $k$ times. To alleviate the computation cost of running $k$ inference passes, we propose Superposed Decoding, a new decoding algorithm that generates $k$ drafts at the computation cost of one autoregressive inference pass. We achieve this by feeding a superposition of the $k$ most recent token embeddings from the drafts as input to the next decoding step of the language model. At every inference step we combine the $k$ drafts with the top-$k$ tokens to get $k^2$ new drafts and cache the $k$ most likely options, using an n-gram interpolation with minimal compute overhead to filter out incoherent generations. Our experiments show that $k$ drafts from Superposed Decoding are at least as coherent and factual as Nucleus Sampling and Greedy Decoding respectively, while being at least $2.44\times$ faster for $k\ge3$. In a compute-normalized setting, user evaluations demonstrably favor text generated by Superposed Decoding over Nucleus Sampling. Code and more examples open-sourced at https://github.com/RAIVNLab/SuperposedDecoding.

翻译：当前许多应用在用户输入时提供多个自动补全草稿，包括GitHub的代码补全、Gmail的智能撰写和苹果消息自动建议。其底层技术通过运行自回归推理过程来提供草稿。因此，向用户提供$k$个草稿需要运行昂贵的语言模型$k$次。为降低$k$次推理过程的计算成本，我们提出叠加解码——一种新的解码算法，能够以单次自回归推理的计算成本生成$k$个草稿。该方法通过将多个草稿中最近$k$个词符嵌入的叠加态作为输入馈送到语言模型的下一个解码步骤来实现。在每个推理步骤中，我们将$k$个现有草稿与top-$k$候选词符组合得到$k^2$个新草稿，并通过最小计算开销的n元插值过滤不连贯生成，缓存$k$个最可能选项。实验表明，叠加解码生成的$k$个草稿在连贯性和事实准确性上至少分别与核采样和贪婪解码相当，且在$k\ge3$时解码速度提升至少$2.44\times$。在计算归一化设定下，用户评估明显更青睐叠加解码生成的文本。代码及更多示例已开源：https://github.com/RAIVNLab/SuperposedDecoding。