We present Ultima, a new collective-communication system for the cloud with bounded, predictable completion times for deep-learning jobs in the presence of varying computation (stragglers) and communication (congestion and gradient drops) variabilities. Ultima exploits the inherent resiliency and the stochastic nature of distributed deep-learning (DDL) training to work with approximated gradients, and provides an efficient balance between (tail) performance and the resulting accuracy of the trained models. Exploiting this domain-specific characteristic of DDL, Ultima introduces (1) mechanisms (e.g., Transpose AllReduce, unreliable connection-oriented transport, and adaptive timeout) to improve the DDL jobs' tail execution time, and (2) strategies (e.g., Hadamard Transform) to mitigate the impact of gradient drops on model accuracy. Our evaluation shows that Ultima achieves 60% faster time-to-accuracy (TTA), on average, when operating in shared environments (e.g., public cloud), and is on par with existing algorithms (e.g., Ring-AllReduce) in dedicated environments (like HPC).

翻译：摘要：本文提出Ultima——一种新型云端集体通信系统，可在存在计算（掉队者）与通信（拥塞与梯度丢失）双重变异性的环境下，为深度学习任务提供有界且可预测的完成时间。Ultima利用分布式深度学习（DDL）训练固有的容错性与随机特性来近似梯度处理，并在（尾延迟）性能与训练模型精度之间实现高效平衡。通过利用DDL的领域特异性，Ultima引入：（1）改善DDL任务尾执行时间的机制（例如：转置AllReduce、不可靠面向连接传输协议及自适应超时策略）；（2）缓解梯度丢失对模型精度影响的策略（例如：哈达玛变换）。实验评估表明，在公有云等共享环境中运行时，Ultima的平均时间-精度（TTA）提升达60%；而在专用环境（如HPC）中，其性能与Ring-AllReduce等现有算法持平。