The transformer's context window is vital for tasks such as few-shot learning and conditional generation as it preserves previous tokens for active memory. However, as the context lengths increase, the computational costs grow quadratically, hindering the deployment of large language models (LLMs) in real-world, long sequence scenarios. Although some recent key-value caching (KV Cache) methods offer linear inference complexity, they naively manage the stored context, prematurely evicting tokens and losing valuable information. Moreover, they lack an optimized prefill/prompt stage strategy, resulting in higher latency than even quadratic attention for realistic context sizes. In response, we introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens, enabling the model to maintain longer context histories without increasing the cache size. Our approach outperforms linear caching baselines across key benchmarks, including streaming perplexity, question answering, book summarization, and passkey retrieval, where it retains better retrieval accuracy at 1M tokens after four doublings of the cache size of 65K. Additionally, our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens. These innovations not only enhance the computational efficiency of LLMs but also pave the way for their effective deployment in resource-constrained environments, enabling large-scale, real-time applications with significantly reduced latency.

翻译：Transformer的上下文窗口对于少样本学习和条件生成等任务至关重要，因为它能够保留先前标记作为活动记忆。然而，随着上下文长度的增加，计算成本呈二次方增长，这阻碍了大型语言模型（LLMs）在现实世界长序列场景中的部署。尽管最近的一些键值缓存（KV Cache）方法提供了线性推理复杂度，但它们对存储上下文的管理方式较为简单，会过早地驱逐标记并丢失有价值的信息。此外，它们缺乏优化的预填充/提示阶段策略，导致在实际上下文长度下的延迟甚至高于二次方注意力机制。为此，我们引入了一种新颖的机制，该机制利用级联子缓存缓冲区有选择地保留最相关的标记，使模型能够在缓存大小不变的情况下维持更长的上下文历史。我们的方法在关键基准测试中均优于线性缓存基线，包括流式困惑度、问答、书籍摘要和密钥检索，在缓存大小从65K经过四次翻倍后，我们的方法在1M标记长度下仍能保持更好的检索准确率。此外，与在1M标记上使用Flash Attention相比，我们的方法将预填充阶段的延迟降低了6.8倍。这些创新不仅提升了LLMs的计算效率，也为它们在资源受限环境中的有效部署铺平了道路，使得大规模、低延迟的实时应用成为可能。