AllReduce is a fundamental collective operation in distributed computing and a key performance bottleneck for large-scale training and inference. Its completion time is determined by the number of communication steps, which dominates latency-sensitive workloads, and the communication distance affecting both latency- and bandwidth-bound regimes. Direct-connect topologies, such as torus networks used in Google's TPUv4, are particularly prone to large communication distances due to limited bisection bandwidth. Latency-optimal algorithms such as Bruck's complete AllReduce in $\log_3 n$ steps on a bidirectional ring, but incur large communication distances that result in substantial congestion. In contrast, recent approaches such as Swing reduce communication distance and congestion, but are inherently required to perform $\log_2 n$ steps to complete AllReduce, sacrificing latency-optimality. In this paper, we present Trivance, a novel AllReduce algorithm that completes within $\log_3 n$ steps, while reducing congestion compared to Bruck's algorithm by a factor of three and preserving bandwidth-optimality. Trivance exploits both transmission ports of a bidirectional ring within each step to triple the communication distance along both directions simultaneously. Furthermore, by performing joint reductions, Trivance improves both the number of steps and network congestion. We further show that Trivance extends naturally to multidimensional torus networks, retaining its latency advantage while achieving performance comparable to bandwidth-optimal algorithms for large messages. Our empirical evaluation shows that Trivance improves state-of-the-art approaches by 5-30% for message sizes up to 8\,MiB, in high-bandwidth settings up to 32MiB and for 3D tori up to 128MiB. Throughout the evaluation, Trivance remains the best-performing latency-optimal algorithm.

翻译：AllReduce是分布式计算中的基础集合操作，也是大规模训练和推理的关键性能瓶颈。其完成时间由通信步骤数（在延迟敏感型工作负载中起主导作用）以及通信距离（同时影响延迟敏感型和带宽受限型场景）共同决定。直接连接拓扑（例如Google TPUv4中使用的环面网络）由于二分带宽有限，特别容易产生较大的通信距离。延迟最优算法（如Bruck算法）在双向环上以$\log_3 n$步完成AllReduce，但会产生较大的通信距离，导致严重的网络拥塞。相比之下，Swing等近期方法虽然减少了通信距离和拥塞，但本质上需要执行$\log_2 n$步才能完成AllReduce，牺牲了延迟最优性。本文提出Trivance，一种新颖的AllReduce算法，可在$\log_3 n$步内完成，同时将拥塞程度较Bruck算法降低三倍并保持带宽最优性。Trivance通过在每一步中同时利用双向环的两个传输端口，使双向通信距离同时扩大三倍。此外，通过执行联合归约操作，Trivance同步优化了步骤数和网络拥塞。我们进一步证明Trivance可自然扩展到多维环面网络，在保持延迟优势的同时，对大消息量场景实现与带宽最优算法相当的性能。实验评估表明：对于不超过8MiB的消息量，在高带宽环境下可达32MiB，在三维环面中可达128MiB，Trivance较现有最优方法的性能提升达5-30%。在所有评估场景中，Trivance始终保持为性能最佳的延迟最优算法。