With the rapid evolution of GPU architectures, the heterogeneity of model training infrastructures is steadily increasing. In such environments, effectively utilizing all available heterogeneous accelerators becomes critical for distributed model training. However, existing frameworks, which are primarily designed for homogeneous clusters, often exhibit significant resource underutilization when deployed on heterogeneous accelerators and networks. In this paper, we present Harp, an automated parallel training framework designed specifically for heterogeneous clusters. Harp introduces a fine-grained planner that efficiently searches a wide space for the inter-operator parallel strategy, enabling Harp to alleviate communication overheads while maintaining balanced loads across heterogeneous accelerators. In addition, Harp implements a heterogeneity-aware 1F1B scheduler that adaptively adjusts the execution timing and ordering of microbatches based on network characteristics, maximizing computation-communication overlap under cross-cluster interconnects while incurring only minimal memory overhead. Our evaluation results show that Harp can deliver 1.3x-1.6x higher performance on heterogeneous clusters than state-of-the-art training frameworks.

翻译：随着GPU架构的快速演进，模型训练基础设施的异构性持续增强。在此类环境中，有效利用所有可用异构加速器对分布式模型训练至关重要。然而，现有主要针对同构集群设计的框架在部署于异构加速器与网络时，往往出现显著的资源利用率不足问题。本文提出Harp——专为异构集群设计的自动化并行训练框架。Harp引入细粒度规划器，可高效搜索算子间并行策略的广阔空间，从而在保持异构加速器负载均衡的同时减轻通信开销。此外，Harp实现了一种异构感知的1F1B调度器，能根据网络特性自适应调整微批次的执行时序与顺序，在跨集群互连场景下最大化计算-通信重叠，且仅产生极小的内存开销。评估结果表明，与当前最优训练框架相比，Harp在异构集群上可实现1.3倍至1.6倍的性能提升。