The linear memory growth of the KV cache poses a significant bottleneck for LLM inference in long-context tasks. Existing static compression methods often fail to preserve globally important information, principally because they overlook the attention drift phenomenon where token significance evolves dynamically. Although recent dynamic retrieval approaches attempt to address this issue, they typically suffer from coarse-grained caching strategies and incur high I/O overhead due to frequent data transfers. To overcome these limitations, we propose HeteroCache, a training-free dynamic compression framework. Our method is built on two key insights: attention heads exhibit diverse temporal heterogeneity, and there is significant spatial redundancy among heads within the same layer. Guided by these insights, HeteroCache categorizes heads based on stability and redundancy. Consequently, we apply a fine-grained weighting strategy that allocates larger cache budgets to heads with rapidly shifting attention to capture context changes, thereby addressing the inefficiency of coarse-grained strategies. Furthermore, we employ a hierarchical storage mechanism in which a subset of representative heads monitors attention shift, and trigger an asynchronous, on-demand retrieval of contexts from the CPU, effectively hiding I/O latency. Finally, experiments demonstrate that HeteroCache achieves state-of-the-art performance on multiple long-context benchmarks and accelerates decoding by up to $3\times$ compared to the original model in the 224K context. Our code will be open-source.

翻译：KV缓存的线性内存增长已成为长上下文任务中LLM推理的关键瓶颈。现有静态压缩方法往往难以保留全局重要信息，主要原因是其忽视了注意力漂移现象——即令牌重要性会动态演变。尽管近期动态检索方法尝试解决此问题，但通常存在缓存策略粒度粗糙、频繁数据传输导致高I/O开销等缺陷。为突破这些限制，我们提出HeteroCache——一种免训练的动态压缩框架。该方法基于两个关键发现：注意力头具有显著的时间异质性，且同层注意力头间存在大量空间冗余。基于此，HeteroCache依据稳定性与冗余度对注意力头进行分类，进而采用细粒度加权策略：为注意力快速漂移的头部分配更大缓存预算以捕捉上下文变化，从而解决粗粒度策略的效率问题。此外，我们设计了分层存储机制：通过代表性头部子集监测注意力漂移，触发从CPU的异步按需上下文检索，有效隐藏I/O延迟。实验表明，HeteroCache在多个长上下文基准测试中达到最先进性能，在224K上下文长度下较原始模型解码速度提升最高达$3\times$。代码将开源发布。