Long-context LLM serving requires offloading KV caches to host-memory and SSDs, but existing mechanisms are not designed for such long contexts. We observe significant inefficiencies in current KV caching in long contexts: high serving latency ~30.7 s, exceeding the typical TTFT requirement of 10 s by more than 3x. Our in-depth analysis explains two major reasons: (1) retrieval is serialized through host-memory and SSD, leaving other host-memory modules and SSDs underutilized, and (2) SSD-based KV retrieval spends 84% of its time in the kernel filesystem rather than actual device access. To address the problems, we propose unified KV pooling, which aggregates multiple host-memory modules and SSDs into a single logical pool and distributes KV caches across devices based on their bandwidth. To eliminate the filesystem overhead, we design KV-passthrough, which bypasses the kernel filesystem and directly accesses SSD-resident KV caches from user space via SPDK. Across evaluations on LLaMA 3.1-8B, GPT-OSS-20B, and Qwen3-30B-A3B, unified KV pooling reduces TTFT in long-contexts ~4.1x over state-of-the-art techniques, all making under 10 s. It also reduces blocked I/O time by up to 23.2x by eliminating filesystem overhead.

翻译：长上下文LLM服务需要将KV缓存卸载到主机内存和SSD，但现有机制并非为如此长的上下文设计。我们观察到当前长上下文KV缓存存在显著低效：服务延迟高达约30.7秒，超过典型TTFT要求（10秒）3倍以上。深入分析揭示了两大主要原因：(1)检索过程在主机内存和SSD之间串行化，导致其他主机内存模块和SSD利用率不足；(2)基于SSD的KV检索中，84%时间消耗在内核文件系统而非实际设备访问。针对这些问题，我们提出统一KV池化技术，将多个主机内存模块和SSD整合为单一逻辑池，并根据设备带宽分布KV缓存。为消除文件系统开销，我们设计了KV直通机制，通过SPDK绕过内核文件系统，直接从用户空间访问SSD中的KV缓存。在LLaMA 3.1-8B、GPT-OSS-20B和Qwen3-30B-A3B上的评估表明，统一KV池化将长上下文的TTFT相比最先进技术降低约4.1倍，所有场景均控制在10秒以内。同时，通过消除文件系统开销，阻塞I/O时间最多减少23.2倍。