Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces. However, as the sequence length grows, the key-value (KV) cache expands linearly, incurring significant memory and computation costs. Existing KV cache eviction methods mitigate this issue by discarding less important KV pairs, but often fail to capture complex KV dependencies, resulting in performance degradation. To better balance efficiency and performance, we introduce ForesightKV, a training-based KV cache eviction framework that learns to predict which KV pairs to evict during long-text generations. We first design the Golden Eviction algorithm, which identifies the optimal eviction KV pairs at each step using future attention scores. These traces and the scores at each step are then distilled via supervised training with a Pairwise Ranking Loss. Furthermore, we formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens. Experiments on AIME2024 and AIME2025 benchmarks of three reasoning models demonstrate that ForesightKV consistently outperforms prior methods under only half the cache budget, while benefiting synergistically from both supervised and reinforcement learning approaches.

翻译：近年来，大型语言模型通过生成长推理轨迹展现出卓越的推理能力。然而，随着序列长度增加，键值缓存呈线性扩张，导致显著的内存与计算开销。现有的键值缓存淘汰方法通过丢弃重要性较低的键值对来缓解此问题，但往往难以捕捉复杂的键值依赖关系，导致性能下降。为更好地平衡效率与性能，本文提出ForesightKV——一种基于训练的键值缓存淘汰框架，通过学习预测长文本生成过程中应淘汰的键值对。我们首先设计Golden Eviction算法，该算法利用未来注意力分数在每一步识别最优淘汰键值对。随后通过监督训练与成对排序损失函数，对这些轨迹及每一步的分数进行知识蒸馏。此外，我们将缓存淘汰建模为马尔可夫决策过程，并应用GRPO算法以缓解低熵词元上语言建模损失的大幅上升。在AIME2024与AIME2025基准测试中对三种推理模型的实验表明，ForesightKV在仅使用半数缓存预算的情况下持续优于现有方法，同时从监督学习与强化学习方法中获得了协同增益。