Distributed synchronized GPU training is commonly used for deep learning. The resource constraint of using a fixed number of GPUs makes large-scale training jobs suffer from long queuing time for resource allocation, and lowers the cluster utilization. Adapting to resource elasticity can alleviate this but often introduces inconsistent model accuracy, due to lacking of capability to decouple model training procedure from resource allocation. We propose EasyScale, an elastic training system that achieves consistent model accuracy under resource elasticity for both homogeneous and heterogeneous GPUs. EasyScale preserves the data-parallel training behaviors strictly, traces the consistency-relevant factors carefully, utilizes the deep learning characteristics for EasyScaleThread abstraction and fast context-switching. To utilize heterogeneous cluster, EasyScale dynamically assigns workers based on the intra-/inter-job schedulers, minimizing load imbalance and maximizing aggregated job throughput. Deployed in an online serving cluster, EasyScale powers the training jobs to utilize idle GPUs opportunistically, improving overall cluster utilization by 62.1.

翻译：分布式同步GPU训练是深度学习的常用方法。固定数量GPU的资源约束导致大规模训练作业因资源分配而经历长时间排队，并降低了集群利用率。适应资源弹性可缓解此问题，但由于缺乏将模型训练过程与资源分配解耦的能力，常导致模型准确率不一致。我们提出EasyScale，一种在资源弹性下实现同质及异质GPU均能保持模型准确率一致的弹性训练系统。EasyScale严格保持数据并行训练行为，细致追踪一致性相关因素，利用深度学习特性实现EasyScale线程抽象与快速上下文切换。为利用异质集群，EasyScale基于作业内/作业间调度器动态分配工作节点，最小化负载失衡并最大化聚合作业吞吐量。部署在在线服务集群后，EasyScale支持训练作业机会性利用空闲GPU，使整体集群利用率提升62.1%。