While large language models (LLMs) excel at handling long-context sequences, they require substantial prefill computation and key-value (KV) cache, which can heavily burden computational efficiency and memory usage in both prefill and decoding stages. Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget. This coupling arises from overlooking the layer-dependent variation of critical context, often leading to accuracy degradation. To address this issue, we introduce FastKV, a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers. FastKV performs full-context computation until a Token-Selective Propagation (TSP) layer, which forwards only the most informative tokens to subsequent layers. From these propagated tokens, FastKV independently selects salient KV entries for caching, thereby decoupling KV budget from the prefill compute reduction based on the TSP decision. This independent control of the TSP rate and KV retention rate enables flexible optimization of efficiency and accuracy. Experimental results show that FastKV achieves speedups of up to 1.82$\times$ in prefill and 2.87$\times$ in decoding compared to the full-context baseline, while matching the accuracy of the baselines that only accelerate the decoding stage. Our code is available at https://github.com/dongwonjo/FastKV.

翻译：尽管大语言模型（LLMs）在处理长上下文序列方面表现出色，但其需要大量的预填充计算和键值（KV）缓存，这可能在预填充和解码阶段对计算效率和内存使用造成沉重负担。近期通过预填充加速压缩KV缓存的研究降低了这一成本，但无意中将预填充计算缩减与解码KV预算绑定。这种耦合源于忽视了关键上下文的层间差异，常导致准确性下降。为解决此问题，我们提出了FastKV，一种KV缓存压缩框架，旨在利用后续层中令牌重要性的稳定性来降低预填充和解码的延迟。FastKV执行全上下文计算直至令牌选择性传播（TSP）层，该层仅将信息量最大的令牌转发至后续层。基于这些传播的令牌，FastKV独立选择显著的KV条目进行缓存，从而基于TSP决策将KV预算与预填充计算缩减解耦。这种对TSP率和KV保留率的独立控制实现了效率与准确性的灵活优化。实验结果表明，与全上下文基线相比，FastKV在预填充阶段实现了最高1.82$\times$的加速，解码阶段实现了最高2.87$\times$的加速，同时与仅加速解码阶段的基线模型保持相当的准确性。我们的代码发布于 https://github.com/dongwonjo/FastKV。