Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by recycling pre-generated model states without running the whole model in multiple steps. Our approach relies on the observation that adjacent tokens in a sequence usually have strong correlations and the next token in a sequence can be reasonably guessed or inferred based on the preceding ones. Experiments and analysis demonstrate the effectiveness of our approach in lowering inference latency, achieving up to 1.4x speedup while preserving high performance.

翻译：现有的大型语言模型必须运行K次才能生成包含K个标记的序列。本文提出RecycleGPT，这是一种通过回收预生成的模型状态而无需在多步中运行整个模型，从而实现快速解码的生成式语言模型。我们的方法基于以下观察：序列中的相邻标记通常具有很强的相关性，并且序列中的下一个标记可以基于前面的标记进行合理猜测或推断。实验和分析证明了我们的方法在降低推理延迟方面的有效性，在保持高性能的同时实现了高达1.4倍的加速。