Pipeline parallelism has been demonstrated to be a remarkable approach to improve throughput for training deep neural networks with billions of parameters over heterogeneous clusters. The 1F1B scheduling plan is a widely adopted strategy for memory and performance optimization, which interchanges the forward and backward stage computations of different micro-batches. On the other hand, a common issue in using the 1F1B scheduling is that stage computation is delayed due to the data transfer when network resources are preempted by other tasks, even with the minimum communication between stages. The exclusive access of these network resources cannot be guaranteed in cloud offerings. We present a general scheduling technique to accommodate pipeline parallelism to preempted network environments at the expense of a certain amount of memory pressure. The core concept is to extend 1F1B schedule scheme to kFkB, which groups k micro-batches, and alternately executes k forward and backward computations. We propose Ada-Grouper, an adaptive kFkB scheduler which regularly adjusts the number of group members k to maintain an optimal balance between communication and computation efficiency correspond to changes in a changing network environment under the memory limit. Experimental results demonstrate that our design maintain stable performance for pipeline parallelism, yielding a performance increase of up from 4% to 30%, compared with 1F1B in preempted network scenarios.

翻译：摘要：流水线并行已被证明是一种在异构集群上训练具有数十亿参数的深度神经网络时提升吞吐量的卓越方法。1F1B调度方案是一种广泛采用的内存与性能优化策略，它交替执行不同微批次的前向和后向阶段计算。然而，使用1F1B调度时常见的问题是，即使阶段间通信量最小化，当网络资源被其他任务抢占时，阶段计算仍会因数据传输而延迟。在云服务中，这些网络资源的独占访问无法得到保证。我们提出了一种通用调度技术，以适应抢占式网络环境下的流水线并行，代价是一定程度的内存压力增大。其核心思想是将1F1B调度方案扩展为kFkB，即将k个微批次分组，并交替执行k次前向和后向计算。我们提出了Ada-Grouper，一种自适应的kFkB调度器，它定期调整组成员数量k，以在内存限制下，根据网络环境变化动态维持通信与计算效率的最优平衡。实验结果表明，在抢占式网络场景中，与1F1B相比，我们的设计能保持流水线并行的稳定性能，性能提升幅度从4%到30%不等。