The NVIDIA Collective Communication Library (NCCL) is a critical software layer enabling high-performance collectives on large-scale GPU clusters. Despite being open source with a documented API, its internal design remains largely opaque. The orchestration of communication channels, selection of protocols, and handling of memory movement across devices and nodes are not well understood, making it difficult to analyze performance or identify bottlenecks. This paper presents a comprehensive analysis of NCCL, focusing on its communication protocol variants (Simple, LL, and LL128), mechanisms governing intra-node and inter-node data movement, and ring- and tree-based collective communication algorithms. The insights obtained from this study serve as the foundation for ATLAHS, an application-trace-driven network simulation toolchain capable of accurately reproducing NCCL communication patterns in large-scale AI training workloads. By demystifying NCCL's internal architecture, this work provides guidance for system researchers and performance engineers working to optimize or simulate collective communication at scale.

翻译：NVIDIA集体通信库（NCCL）是实现大规模GPU集群高性能集体操作的关键软件层。尽管其作为开源项目提供了文档化的API，但其内部设计在很大程度上仍不透明。通信通道的协调、协议选择以及跨设备和节点的内存移动处理机制尚未得到充分理解，这导致性能分析与瓶颈识别存在困难。本文对NCCL进行了全面分析，重点研究其通信协议变体（Simple、LL和LL128）、管理节点内与节点间数据移动的机制，以及基于环状与树状的集体通信算法。本研究获得的洞见构成了ATLAHS的基础——这是一个能够准确复现大规模AI训练工作负载中NCCL通信模式的应用轨迹驱动网络模拟工具链。通过揭示NCCL的内部架构，本工作为致力于大规模集体通信优化或模拟的系统研究人员与性能工程师提供了指导。