Host-GPU data movement has become a latency-critical bottleneck in LLM serving, surfacing in common paths such as model-weight movement and KV cache offload/fetch. Today, each host-GPU copy is effectively confined to the PCIe path of the target GPU, even though modern multi-GPU servers contain additional PCIe links on peer GPUs and high bandwidth GPU interconnects. This leaves substantial intra-server I/O capacity unused. To address this issue, we present Multipath Memory Access (MMA), a software-defined multipath memory access system for host--GPU data transfer. To the best of our knowledge, MMA is the first software-defined system to enable efficient multipath host--GPU data transfer within a single multi-GPU server. MMA expands a single host--GPU copy across available direct and relay paths without hardware, driver, or application changes. It preserves CUDA stream semantics with a dependency-preserving Dummy Task, coordinates distributed micro-transfer completion through a lightweight synchronization mechanism, and uses queue backpressure to route traffic without explicit link-state feedback. On an 8-GPU NVIDIA H20 server, MMA achieves 245 GB/s peak host-to-GPU bandwidth, a 4.62x improvement over native CUDA copies, and reduces TTFT for KV cache fetching by 1.14-2.38x and model wake-up/switching latency by 1.12-2.48x.

翻译：主机与GPU之间的数据传输已成为LLM服务中延迟敏感的关键瓶颈，常见于模型权重移动和KV缓存卸载/加载等场景。当前，即使现代多GPU服务器包含对等GPU上的额外PCIe链路和高带宽GPU互连，每次主机到GPU的拷贝实际上仍局限于目标GPU的PCIe路径，导致大量服务器内部I/O容量未被利用。为解决此问题，我们提出多路径内存访问（MMA）——一种用于主机-GPU数据传输的软件定义多路径内存访问系统。据我们所知，MMA是首个在单台多GPU服务器内实现高效多路径主机-GPU数据传输的软件定义系统。MMA通过利用可用的直连路径和中继路径扩展单次主机-GPU拷贝，无需更改硬件、驱动程序或应用程序。它通过依赖保持的虚拟任务（Dummy Task）保留CUDA流语义，借助轻量级同步机制协调分布式微传输完成，并利用队列背压机制在不依赖显式链路状态反馈的情况下进行流量路由。在8-GPU NVIDIA H20服务器上，MMA实现了245 GB/s的峰值主机到GPU带宽，相较原生CUDA拷贝提升4.62倍，并将KV缓存获取的首次令牌时间（TTFT）降低1.14-2.38倍，模型唤醒/切换延迟降低1.12-2.48倍。