Large deep learning models have shown great potential for delivering exceptional results in various applications. However, the training process can be incredibly challenging due to the models' vast parameter sizes, often consisting of hundreds of billions of parameters. Common distributed training methods, such as data parallelism, tensor parallelism, and pipeline parallelism, demand significant data communication throughout the process, leading to prolonged wait times for some machines in physically distant distributed systems. To address this issue, we propose a novel solution called Hulk, which utilizes a modified graph neural network to optimize distributed computing systems. Hulk not only optimizes data communication efficiency between different countries or even different regions within the same city, but also provides optimal distributed deployment of models in parallel. For example, it can place certain layers on a machine in a specific region or pass specific parameters of a model to a machine in a particular location. By using Hulk in experiments, we were able to improve the time efficiency of training large deep learning models on distributed systems by more than 20\%. Our open source collection of unlabeled data:https://github.com/DLYuanGod/Hulk.

翻译：大型深度学习模型在各类应用中展现出了卓越的性能潜力。然而，由于模型参数量极为庞大（常达数千亿级），其训练过程面临巨大挑战。常见的分布式训练方法（如数据并行、张量并行、流水线并行）在整个训练流程中需要频繁的数据通信，导致物理距离较远的分布式系统中部分机器出现长时间等待。为解决这一问题，我们提出了一种名为Hulk的创新方案，采用改进的图神经网络对分布式计算系统进行优化。Hulk不仅能优化跨国家乃至同城市不同区域间的数据通信效率，还能实现模型在并行计算中的最优分布式部署。例如，它可将特定层部署于某一区域的机器上，或将模型的特定参数传递至特定地理位置的机器。实验表明，通过使用Hulk，分布式系统上大型深度学习模型的训练时间效率提升了20%以上。我们已开源无标注数据集合：https://github.com/DLYuanGod/Hulk。