Reasoning large language models exhibit complex reasoning behaviors via extended chain-of-thought generation that are highly fragile to information loss during decoding, creating critical challenges for KV cache compression. Existing token-dropping methods directly disrupt reasoning chains by removing intermediate steps, while head-reallocation methods, designed for retrieval tasks, fail to preserve the heads essential for generative reasoning. However, no existing method can identify which attention heads genuinely maintain reasoning consistency and control generation termination. To address this, we propose RLKV, which uses reinforcement learning as a probe to discover which heads contribute to reasoning quality by directly optimizing their cache usage against actual generation outcomes. This discovery naturally leads to an efficient compression strategy: we allocate full KV cache to reasoning-critical heads while aggressively compressing others with constant-size KV cache. Experiments reveal that a fraction of heads proves essential for reasoning, enabling 20--60% cache reduction with near-lossless performance across diverse tasks and models, and up to 2.06x end-to-end speedup at 60% reduction.

翻译：推理型大语言模型通过扩展的思维链生成展现复杂的推理行为，但其解码过程中对信息损失高度敏感，这为KV缓存压缩带来了关键挑战。现有的词元丢弃方法直接移除中间步骤，会破坏推理链；而面向检索任务设计的头部重分配方法，无法保留生成式推理所必需的注意力头。然而，当前尚无方法能识别哪些注意力头真正维持推理连贯性并控制生成终止。为解决这一问题，我们提出RLKV方法，利用强化学习作为探针，通过直接优化注意力头的缓存使用对实际生成结果的影响，来发现哪些头部对推理质量有贡献。这一发现自然引出了高效的压缩策略：为推理关键头部分配完整KV缓存，同时对其他头部采用固定大小的KV缓存进行激进压缩。实验表明，仅需少量头部即可保证推理效果，在多种任务和模型上实现20%-60%的缓存压缩且性能近乎无损，在60%压缩率下端到端加速比可达2.06倍。