Communication efficiency plays an important role in accelerating the distributed training of Deep Neural Networks (DNN). All-reduce is the crucial communication primitive to reduce model parameters in distributed DNN training. Most existing all-reduce algorithms are designed for traditional electrical interconnect systems, which cannot meet the communication requirements for distributed training of large DNNs due to the low data bandwidth of the electrical interconnect systems. One of the promising alternatives for electrical interconnect is optical interconnect, which can provide high bandwidth, low transmission delay, and low power cost. We propose an efficient scheme called WRHT (Wavelength Reused Hierarchical Tree) for implementing all-reduce operation in optical interconnect systems. WRHT can take advantage of WDM (Wavelength Division Multiplexing) to reduce the communication time of distributed data-parallel DNN training. We further derive the required number of wavelengths, the minimum number of communication steps, and the communication time for the all-reduce operation on optical interconnect. The constraint of insertion loss is also considered in our analysis. Simulation results show that the communication time of all-reduce by WRHT is reduced by 80.81%, 64.36%, and 82.12%, respectively, compared with three traditional all-reduce algorithms according to our simulation results of an optical interconnect system. Our results also show that WRHT can reduce the communication time of all-reduce operation by 92.42% and 91.31% compared to two existing all-reduce algorithms running in the electrical interconnect system.

翻译：通信效率在加速深度神经网络（DNN）分布式训练中扮演着重要角色。全规约（All-reduce）是分布式DNN训练中用于归约模型参数的关键通信原语。现有的大多数全规约算法专为传统电互连系统设计，由于电互连系统数据带宽较低，无法满足大规模DNN分布式训练的通信需求。光学互连作为电互连的有前景替代方案，能够提供高带宽、低传输延迟和低功耗成本。我们提出一种名为WRHT（波长复用分层树）的高效方案，用于在光学互连系统中实现全规约操作。WRHT可利用波分复用（WDM）技术减少分布式数据并行DNN训练的通信时间。我们进一步推导了光学互连中全规约操作所需的波长数、最小通信步骤数及通信时间，并在分析中考虑了插入损耗约束。仿真结果表明，在光学互连系统下，与三种传统全规约算法相比，WRHT的全规约通信时间分别降低了80.81%、64.36%和82.12%。同时，与电互连系统中运行的两种现有全规约算法相比，WRHT可将全规约操作的通信时间降低92.42%和91.31%。