Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for long-context LLMs poses significant deployment challenges. Existing approaches either discard potentially critical information needed for future generations or offer limited efficiency gains due to high computational overhead. In this paper, we introduce CentroidKV, a simple yet effective framework for online KV cache clustering. Our approach is based on the observation that key states exhibit high similarity along the sequence dimension. To enable efficient clustering, we divide the sequence into chunks and propose Chunked Soft Matching, which employs an alternating partition strategy within each chunk and identifies clusters based on similarity. CentroidKV then merges the KV cache within each cluster into a single centroid. Additionally, we provide a theoretical analysis of the computational complexity and the optimality of the intra-chunk partitioning strategy. Extensive experiments across various models and long-context benchmarks demonstrate that CentroidKV achieves up to 75% reduction in KV cache memory usage while maintaining comparable model performance. Moreover, with minimal computational overhead, CentroidKV accelerates the decoding stage of inference by up to $1.92\times$ and increases the serving throughput by up to $4\times$.

翻译：具有扩展上下文窗口的大型语言模型（LLMs）在应对复杂任务中日益普及。然而，长上下文LLM所需的大量键值（KV）缓存给实际部署带来了显著挑战。现有方法要么丢弃未来生成中可能的关键信息，要么因高计算开销而增益有限。本文提出CentroidKV——一种简单而有效的在线KV缓存聚类框架。我们的方法基于观察：键状态在序列维度上表现出高度相似性。为实现高效聚类，我们将序列划分为块并提出分块软匹配（Chunked Soft Matching）策略，该策略在每个块内采用交替分区方案，基于相似性识别聚类簇。CentroidKV随后将每个簇内的KV缓存合并为单一质心。此外，我们提供了计算复杂度与块内分区策略最优性的理论分析。在多种模型和长上下文基准上的广泛实验表明，CentroidKV在维持相当模型性能的同时，最高可减少75%的KV缓存内存占用。更重要的是，在极低计算开销下，CentroidKV将推理解码阶段加速最高达$1.92\times$，并使服务吞吐量提升最高达$4\times$。