While large language models (LLMs) excel at handling long-context sequences, they require substantial prefill computation and key-value (KV) cache, which can heavily burden computational efficiency and memory usage in both prefill and decoding stages. Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget. This coupling arises from overlooking the layer-dependent variation of critical context, often leading to accuracy degradation. To address this issue, we introduce FastKV, a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers. FastKV performs full-context computation until a Token-Selective Propagation (TSP) layer, which forwards only the most informative tokens to subsequent layers. From these propagated tokens, FastKV independently selects salient KV entries for caching, thereby decoupling KV budget from the prefill compute reduction based on the TSP decision. This independent control of the TSP rate and KV retention rate enables flexible optimization of efficiency and accuracy. Experimental results show that FastKV achieves speedups of up to 1.82$\times$ in prefill and 2.87$\times$ in decoding compared to the full-context baseline, while matching the accuracy of the baselines that only accelerate the decoding stage. Our code is available at https://github.com/dongwonjo/FastKV.

翻译：尽管大语言模型（LLMs）在处理长上下文序列方面表现出色，但其需要大量的预填充计算和键值（KV）缓存，这会在预填充和解码阶段对计算效率和内存使用造成沉重负担。近期通过预填充加速来压缩KV缓存的研究降低了这一成本，但无意中将预填充计算缩减与解码KV预算绑定在一起。这种耦合源于忽略了关键上下文的层间差异性，常常导致精度下降。为解决这一问题，我们提出了FastKV，这是一个KV缓存压缩框架，旨在通过利用后续层中令牌重要性的稳定性来降低预填充和解码的延迟。FastKV执行全上下文计算直至令牌选择性传播（TSP）层，该层仅将信息量最大的令牌前向传播至后续层。基于这些传播的令牌，FastKV独立地选择显著的KV条目进行缓存，从而基于TSP决策将KV预算与预填充计算缩减解耦。这种对TSP速率和KV保留率的独立控制，使得效率和精度能够灵活优化。实验结果表明，与全上下文基线相比，FastKV在预填充阶段实现了高达1.82倍的加速，在解码阶段实现了高达2.87倍的加速，同时匹配了仅加速解码阶段的基线模型的精度。我们的代码可在 https://github.com/dongwonjo/FastKV 获取。