As large language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce \textbf{PiKV}, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages \textit{expert-sharded KV storage} to partition caches across GPUs, \textit{PiKV routing} to reduce token-to-KV access, and a \textit{PiKV Scheduling} to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates \textit{PiKV Compression} modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: \href{https://github.com/NoakLiu/PiKV}{https://github.com/NoakLiu/PiKV}. Experiments details is recorded at: \href{https://github.com/NoakLiu/PiKV/blob/main/downstream_tasks/README.md}{https://github.com/NoakLiu/PiKV/Experimental\_Results}. We also have PiKV integrated with Nvidia kvpress for acceleration, details see \href{https://github.com/NoakLiu/PiKVpress}{https://github.com/NoakLiu/PiKVpress}. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.

翻译：随着大语言模型在规模和上下文长度上的持续扩展，键值（KV）缓存存储的内存与通信开销已成为多GPU与多节点推理的主要瓶颈。尽管基于专家混合（MoE）的架构通过专家实现了计算稀疏化，但相应的KV缓存仍保持密集且全局同步，导致显著的开销。本文提出**PiKV**，一个专为MoE架构设计的并行分布式KV缓存服务框架。PiKV采用**专家分片KV存储**将缓存分区至各GPU，利用**PiKV路由**减少令牌对KV的访问，并通过**PiKV调度**自适应地保留与查询相关的条目。为进一步降低内存使用，PiKV将**PiKV压缩**模块集成至缓存流水线以加速处理。PiKV近期已作为开源软件库公开发布：https://github.com/NoakLiu/PiKV。实验细节记录于：https://github.com/NoakLiu/PiKV/blob/main/downstream_tasks/README.md。我们还将PiKV与Nvidia kvpress集成以加速，详情参见https://github.com/NoakLiu/PiKVpress。PiKV仍是一个持续开发的项目，旨在成为面向MoE架构的综合性KV缓存管理系统。