Large Language Models (LLMs), despite their remarkable performance across a wide range of tasks, necessitate substantial GPU memory and consume significant computational resources. Beyond the memory taken up by model weights, the memory used by the KV cache rises linearly with sequence length, becoming a primary bottleneck for inference. In this paper, we introduce an innovative method for optimizing the KV cache, which considerably minimizes its memory footprint. Upon thorough investigation, we discover that in most Transformer models, (i) there is a striking similarity between adjacent tokens' query vectors, and (ii) the attention calculation of the current query can rely exclusively on the attention information of a small fraction of preceding queries. Based on these observations, we present CORM, a KV cache eviction policy that dynamically retains essential key-value pairs for inference without the need for model fine-tuning. Our validation shows that CORM reduces the inference memory usage of KV cache by up to 70\% with negligible performance degradation across six tasks in LongBench. Furthermore, we demonstrate that CORM is compatible with GQA for further compression rate.

翻译：尽管大语言模型（LLM）在广泛任务中展现出卓越性能，但其需要大量GPU内存并消耗显著的计算资源。除了模型权重所占用的内存外，KV缓存使用的内存随序列长度线性增长，成为推理过程的主要瓶颈。本文提出一种创新的KV缓存优化方法，可显著减少其内存占用。通过深入研究，我们发现大多数Transformer模型中存在以下现象：(i) 相邻令牌的查询向量具有高度相似性；(ii) 当前查询的注意力计算可以仅依赖少量先前查询的注意力信息。基于这些观察，我们提出了CORM——一种无需模型微调即可动态保留推理所需关键键值对的KV缓存淘汰策略。验证结果表明，在LongBench的六项任务中，CORM将KV缓存的推理内存使用量降低高达70%，且性能下降可忽略不计。此外，我们证明CORM可与GQA兼容以实现更高的压缩率。