Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by recycling pre-generated model states without running the whole model in multiple steps. Our approach relies on the observation that adjacent tokens in a sequence usually have strong correlations and the next token in a sequence can be reasonably guessed or inferred based on the preceding ones. Through theoretical evaluations and practical tests on downstream text generation tasks, we demonstrate the effectiveness of our approach in lowering inference latency, achieving up to 1.4x speedup while preserving high performance.

翻译：现有的大语言模型需要运行K次才能生成包含K个令牌的序列。本文提出了RecycleGPT，一种通过回收预生成的模型状态实现快速解码的生成式语言模型，无需多次运行完整模型。该方法基于以下观察：序列中相邻令牌通常具有强相关性，且序列中的下一个令牌可根据前序令牌进行合理推测。通过理论评估与下游文本生成任务的实践测试，我们验证了该方法在降低推理延迟方面的有效性，能够在保持高性能的同时实现高达1.4倍的加速。