Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by recycling pre-generated model states without running the whole model in multiple steps. Our approach relies on the observation that adjacent tokens in a sequence usually have strong correlations and the next token in a sequence can be reasonably guessed or inferred based on the preceding ones. Experiments and analysis demonstrate the effectiveness of our approach in lowering inference latency, achieving up to 1.4x speedup while preserving high performance.

翻译：现有的大语言模型需要运行K次才能生成K个Token序列。本文提出RecycleGPT，一种通过回收预先生成的模型状态实现快速解码的生成式语言模型，无需以多步方式运行完整模型。该方法基于如下观察：序列中相邻Token通常具有强相关性，且序列中的下一个Token可以基于前序Token进行合理推测或推断。实验与分析表明，本方法在保持高性能的同时，可将推理延迟降低至1.4倍速提升。