Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.

翻译：混合专家（Mixture-of-Experts, MoE）架构已成为扩展大型语言模型的关键技术，推动了如DeepEP、Hybrid-EP等专用设备端通信库的发展。这些库展示了GPU发起的RDMA技术在MoE分发与合并操作中的性能优势。本文提出NCCL EP（Expert Parallelism），一个完全基于NCCL设备API构建的从零设计的MoE通信库。NCCL EP提供了统一的ncclEpDispatch和ncclEpCombine原语，支持C和Python接口，并提供面向推理解码的低延迟（Low-Latency, LL）模式和面向训练及推理预填充的高吞吐（High-Throughput, HT）模式。LL模式针对小批量（1-128个token），采用直接全互联RDMA+NVLink网格拓扑，通过双缓冲通信实现分发与合并阶段的重叠。HT模式针对大批量（4096个token以上），采用分层通信策略，在节点间RDMA传输前先聚合NVLink域内的token。两种模式均利用设备API实现节点内和节点间通信，充分发挥其拓扑感知与优化的GPU发起的实现优势。我们在基于H100集群的多节点配置上对NCCL EP进行了评估，展示了具有竞争力的LL内核性能，并给出了与vLLM集成的端到端测试结果。通过将MoE通信原生构建于NCCL框架内，NCCL EP为当前及新兴NVIDIA平台上专家并行的实现提供了一条受支持的路径。