DNN models are becoming increasingly larger to achieve unprecedented accuracy, and the accompanying increased computation and memory requirements necessitate the employment of massive clusters and elaborate parallelization strategies to accelerate DNN training. In order to better optimize the performance and analyze the cost, it is indispensable to model the training throughput of distributed DNN training. However, complex parallelization strategies and the resulting complex runtime behaviors make it challenging to construct an accurate performance model. In this paper, we present Proteus, the first standalone simulator to model the performance of complex parallelization strategies through simulation execution. Proteus first models complex parallelization strategies with a unified representation named Strategy Tree. Then, it compiles the strategy tree into a distributed execution graph and simulates the complex runtime behaviors, comp-comm overlap and bandwidth sharing, with a Hierarchical Topo-Aware Executor (HTAE). We finally evaluate Proteus across a wide variety of DNNs on three hardware configurations. Experimental results show that Proteus achieves $3.0\%$ average prediction error and preserves order for training throughput of various parallelization strategies. Compared to state-of-the-art approaches, Proteus reduces prediction error by up to $133.8\%$.

翻译：深度神经网络模型的规模日益增大以追求前所未有的精度，随之而来的计算与内存需求激增，促使研究者采用大规模集群和精细化的并行策略来加速训练过程。为更好地优化性能并分析成本，对分布式DNN训练吞吐量进行建模不可或缺。然而，复杂的并行策略及其引发的运行时行为使得构建精确性能模型极具挑战。本文提出Proteus——首个通过模拟执行来建模复杂并行策略性能的独立仿真器。Proteus首先以统一表示"策略树"（Strategy Tree）对复杂并行策略进行建模；随后将策略树编译为分布式执行图，并借助层级拓扑感知执行器（HTAE）模拟包含计算-通信重叠与带宽共享的复杂运行时行为。我们最终在三种硬件配置上对涵盖多种DNN的模型进行评测。实验结果表明，Proteus的平均预测误差为3.0%，并能保持各并行策略训练吞吐量的排序一致性。与现有最优方法相比，Proteus将预测误差降低了高达133.8%。