The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets.

翻译：大语言模型（LLM）在边缘AI系统上的推理部署日益增长，亟需在严格内存预算下实现高效执行。关键挑战源于键值（KV）缓存，其规模往往超出可用设备内存。尽管基于NVMe的卸载方案可提供可扩展容量，但现有基于文件的设计高度依赖内核页缓存，在内存压力下会导致缓存抖动、不可预测的延迟以及高昂的软件开销。我们提出双刃（DUAL-BLADE），一种双路径KV驻留框架，能够根据运行时内存可用性动态地将KV张量分配给页缓存路径或NVMe直通路径。NVMe直通路径通过将KV张量映射到连续逻辑块地址（LBA）区域来绕过文件系统，实现低开销的直接存储访问。双刃进一步融入自适应流水线并行机制，使存储I/O与GPU DMA重叠，从而提升推理吞吐量。评估表明，双刃显著缓解了I/O瓶颈，将预填充和解码延迟分别降低最高33.1%和42.4%，并在不同内存预算下将SSD利用率提升2.2倍。