Joint consideration of scheduling and adaptive parallelism offers great opportunities for improving the training efficiency of large models on heterogeneous GPU clusters. However, integrating adaptive parallelism into a cluster scheduler expands the cluster scheduling space. The new space is the product of the original scheduling space and the parallelism exploration space of adaptive parallelism (also a product of pipeline, data, and tensor parallelism). The exponentially enlarged scheduling space and ever-changing optimal parallelism plan from adaptive parallelism together result in the contradiction between low-overhead and accurate performance data acquisition for efficient cluster scheduling. This paper presents Crius, a training system for efficiently scheduling multiple large models with adaptive parallelism in a heterogeneous cluster. Crius proposes a novel scheduling granularity called Cell. It represents a job with deterministic resources and pipeline stages. The exploration space of Cell is shrunk to the product of only data and tensor parallelism, thus exposing the potential for accurate and low-overhead performance estimation. Crius then accurately estimates Cells and efficiently schedules training jobs. When a Cell is selected as a scheduling choice, its represented job runs with the optimal parallelism plan explored. Experimental results show that Crius reduces job completion time by up to 48.9% and schedules large models with up to 1.49x cluster throughput improvement.

翻译：调度与自适应并行化的联合考虑为在异构GPU集群上提升大模型训练效率提供了重要机遇。然而，将自适应并行化集成到集群调度器会扩大集群调度空间。该新空间是原始调度空间与自适应并行化探索空间（亦是流水线并行、数据并行和模型并行三者乘积）的笛卡尔积。指数级增大的调度空间与自适应并行化持续变化的最优并行方案共同导致高效集群调度的低开销与精确性能数据采集之间的矛盾。本文提出Crius——一种在异构集群中通过自适应并行化高效调度多个大模型的训练系统。Crius提出名为Cell的新型调度粒度，每个Cell代表具有确定性资源和流水线阶段的作业。Cell的探索空间被缩减为数据并行与模型并行的乘积，因此具有精确且低开销性能估计的潜力。Crius据此精确估计Cell并高效调度训练任务。当Cell被选为调度决策后，其表征的作业将以探索获得的最优并行方案运行。实验结果表明，Crius可将作业完成时间降低最高48.9%，并实现大模型集群吞吐量提升达1.49倍。