In long-context Large Language Model (LLM) inference, the Time-To-First-Token (TTFT) latency incurred by the prefill stage has become the foremost bottleneck limiting interactive performance and deployment cost. KV Cache reuse offers a direct path to reduce redundant prefill, yet traditional prefix caching applies only to strict-prefix scenarios; directly reusing KV Cache in non-prefix settings breaks the cross-chunk global attention relationships and causes significant degradation in generation quality. When reusable KV Cache is offloaded to GPU-external cache pools, I/O overheads across heterogeneous hardware tiers further emerge as a new TTFT bottleneck. Efficient non-prefix KV Cache reuse therefore requires both semantic-consistency recovery and compute-I/O co-optimization. This paper presents CacheTune, a frequency-guided and hardware-aware KV Cache reuse system for long-context LLM serving. CacheTune first identifies, offline, the KV pairs most critical to cross-attention recovery through frequency-domain analysis, and then selectively recomputes only these semantic-critical tokens online while reusing the remaining KVs. To turn this semantic selection into end-to-end latency reduction, CacheTune further combines sparse KV transfer, multi-stream asynchronous overlap, deferred positional-encoding recovery, and hardware-aware adaptive recomputation-ratio tuning to balance computation and data movement across heterogeneous cache pools. Evaluations on mainstream LLMs and long-context tasks show that CacheTune achieves 3.72x-4.86x TTFT speedup and 3.93x-6.21x higher throughput while maintaining generation quality close to full recompute. Even when caches are offloaded to I/O-bound SSD/HDD storage, CacheTune sustains 2.34x-2.36x TTFT speedup through adaptive recomputation.

翻译：在长上下文大语言模型（LLM）推理中，预填充阶段产生的首个令牌生成时间（TTFT）延迟已成为制约交互性能和部署成本的首要瓶颈。KV缓存重用可直接减少冗余预填充，但传统前缀缓存仅适用于严格前缀场景；在非前缀场景中直接重用KV缓存会破坏跨片段全局注意力关系，导致生成质量显著下降。当可重用KV缓存被卸载至GPU外部缓存池时，跨异构硬件层级间的I/O开销进一步成为新的TTFT瓶颈。高效的非前缀KV缓存重用因此需要同时实现语义一致性恢复和计算-I/O协同优化。本文提出CacheTune——一种面向长上下文LLM服务的频率导向且硬件感知的KV缓存重用系统。CacheTune首先通过频域分析离线识别对跨注意力恢复至关重要的KV键值对，随后仅在线选择性重新计算这些语义关键令牌，同时重用其余KV。为将语义选择转化为端到端延迟降低，CacheTune进一步结合稀疏KV传输、多流异步重叠、延迟位置编码恢复及硬件感知的自适应重计算比率调优，以平衡异构缓存池间的计算与数据移动。在主流LLM和长上下文任务上的评估表明，CacheTune实现了3.72倍-4.86倍的TTFT加速和3.93倍-6.21倍的吞吐量提升，同时保持生成质量接近完全重计算水平。即使缓存被卸载至I/O受限的SSD/HDD存储，CacheTune通过自适应重计算仍可维持2.34倍-2.36倍的TTFT加速。