This paper aims to answer the question: Can deep learning models be cost-efficiently trained on a global market of spot VMs spanning different data centers and cloud providers? To provide guidance, we extensively evaluate the cost and throughput implications of training in different zones, continents, and clouds for representative CV, NLP, and ASR models. To expand the current training options further, we compare the scalability potential for hybrid-cloud scenarios by adding cloud resources to on-premise hardware to improve training throughput. Finally, we show how leveraging spot instance pricing enables a new cost-efficient way to train models with multiple cheap VMs, trumping both more centralized and powerful hardware and even on-demand cloud offerings at competitive prices.

翻译：本文旨在回答以下问题：能否在全球范围内跨越不同数据中心和云服务提供商，利用竞价虚拟机经济高效地训练深度学习模型？为指导实践，我们针对代表性的计算机视觉、自然语言处理和自动语音识别模型，全面评估了在不同可用区、大洲和云平台进行训练的成本与吞吐量影响。为拓展现有训练方案，我们通过将云资源与本地硬件结合以提升训练吞吐量，对比了混合云场景的可扩展潜力。最后，我们论证了利用竞价实例定价机制，能够通过多台廉价虚拟机实现经济高效的模型训练新范式，其成本效益优于集中式高性能硬件，甚至可与具有竞争力的按需云服务方案相抗衡。