Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs, such as LLM-driven agents. However, existing LLMs, pre-trained on sequences with restricted maximum length, cannot generalize to longer sequences due to the out-of-domain and distraction issues. To alleviate these issues, existing efforts employ sliding attention windows and discard distant tokens to achieve the processing of extremely long sequences. Unfortunately, these approaches inevitably fail to capture long-distance dependencies within sequences to deeply understand semantics. This paper introduces a training-free memory-based method, InfLLM, to unveil the intrinsic ability of LLMs to process streaming long sequences. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences while maintaining the ability to capture long-distance dependencies. Without any training, InfLLM enables LLMs pre-trained on sequences of a few thousand tokens to achieve superior performance than competitive baselines continually training these LLMs on long sequences. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies.

翻译：大语言模型（LLM）已成为涉及长流式输入的真实世界应用（如LLM驱动的智能体）的基石。然而，现有LLM在受限最大长度的序列上预训练后，因分布外和注意力分散问题而无法泛化至更长序列。为缓解这些问题，现有研究采用滑动注意力窗口并丢弃远端标记以处理超长序列。然而，这些方法不可避免地无法捕获序列中的长距离依赖关系以深入理解语义。本文提出一种基于无训练记忆的方法InfLLM，以揭示LLM处理流式长序列的内在能力。具体而言，InfLLM将远端上下文存储至额外记忆单元，并采用高效机制查找与标记相关的单元以进行注意力计算。由此，InfLLM使LLM能高效处理长序列，同时保持捕获长距离依赖的能力。无需任何训练，InfLLM即可使在数千标记序列上预训练的LLM获得优于持续在长序列上训练这些LLM的竞争基线的性能。即使序列长度扩展至$1,024$K，InfLLM仍能有效捕获长距离依赖。