Remote KV cache reuse fetches KV cache for identical contexts from remote storage, avoiding recomputation, accelerating LLM inference. While it excels in high-speed networks, its performance degrades significantly in bandwidth-limited scenarios. Recent studies address this by transmitting KV caches in compressed form, but the associated heavyweight decompression counteracts the KV reuse benefits. In this paper, we propose an efficient and widely deployable remote KV cache reuse solution that leverages GPU-native video codecs. Our system, KVFetcher, enables effective KV cache coding with two techniques. The codec-friendly tensor layout compresses the KV cache in a highly compact video format, enabling fast transmission. The efficient KV fetcher orchestrates the transmission, decoding, and restoration of compressed KV caches in an efficient pipelined manner, eliminating resource contention, masking network fluctuations, and achieving minimum time-to-first-token (TTFT). We prototype KVFetcher on diverse GPUs from high- to low-end. Experiments reveal that it reduces TTFT by up to 3.51 times while maintaining lossless accuracy, compared to SOTA methods.

翻译：远程键值缓存复用通过从远程存储获取相同上下文的键值缓存，避免了重复计算，从而加速大语言模型推理。尽管该方法在高速网络中表现优异，但在带宽受限场景下性能显著下降。近期研究通过传输压缩形式的键值缓存来解决此问题，但相关的重量级解压操作抵消了键值复用的优势。本文提出一种高效且广泛可部署的远程键值缓存复用方案，该方案利用GPU原生视频编解码器。我们的系统KVFetcher通过两项技术实现有效的键值缓存编码：编解码器友好的张量布局将键值缓存压缩为高度紧凑的视频格式，实现快速传输；高效的键值获取器以流水线方式协调压缩键值缓存的传输、解码与恢复，消除资源争用，掩盖网络波动，并实现最短的首令牌生成时间。我们在从高端到低端的多种GPU上对KVFetcher进行原型实现。实验表明，与现有最优方法相比，该系统在保持无损精度的同时，可将首令牌生成时间最多降低3.51倍。