This paper aims to answer the question: Can deep learning models be cost-efficiently trained on a global market of spot VMs spanning different data centers and cloud providers? To provide guidance, we extensively evaluate the cost and throughput implications of training in different zones, continents, and clouds for representative CV, NLP, and ASR models. To expand the current training options further, we compare the scalability potential for hybrid-cloud scenarios by adding cloud resources to on-premise hardware to improve training throughput. Finally, we show how leveraging spot instance pricing enables a new cost-efficient way to train models with multiple cheap VMs, trumping both more centralized and powerful hardware and even on-demand cloud offerings at competitive prices.

翻译：本文旨在回答以下问题：能否在覆盖不同数据中心和云提供商的全球竞价虚拟机市场上，以成本高效的方式训练深度学习模型？为提供指导，我们广泛评估了在多个区域、大洲和云平台上训练代表性计算机视觉、自然语言处理和自动语音识别模型时的成本与吞吐量影响。为进一步拓展当前训练选项，我们对比了通过向本地硬件添加云资源来提升训练吞吐量的混合云场景可扩展性潜力。最后，我们展示了如何利用竞价实例定价机制，通过多台低成本虚拟机以具有竞争力的价格实现一种新的成本高效训练方式，其表现优于更集中化的强算力硬件甚至按需云服务。