Deep learning (DL) has become a key component of modern software. In the "big model" era, the rich features of DL-based software substantially rely on powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which are trained on the powerful cloud with large datasets. Hence, training effective DL models has become a vital stage in the whole software lifecycle. When training deep learning models, especially those big models, developers need to parallelize and distribute the computation and memory resources amongst multiple devices in the training process, which is known as distributed deep learning training, or distributed training for short. However, the unique challenges that developers encounter in distributed training process have not been studied in the software engineering community. Given the increasingly heavy dependence of current DL-based software on distributed training, this paper aims to fill in the knowledge gap and presents the first comprehensive study on developers' issues in distributed training. To this end, we analyze 1,131 real-world developers' issues about using these frameworks reported on Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. Based on the results, we suggest actionable implications on research avenues that can potentially facilitate the distributed training to develop DL-based software, such as focusing on the frequent and common fix patterns when designing testing or debugging tools, developing efficient testing and debugging techniques for communication configuration along with the synthesis of network configuration analysis, designing new multi-device checkpoint-and-replay techniques to help reproduction, and designing serverless APIs for cloud platforms.

翻译：深度学习已成为现代软件的关键组成部分。在"大模型"时代，基于深度学习的软件之丰富功能主要依赖于强大的深度学习模型（如BERT、GPT-3以及近期出现的GPT-4），这些模型需要在配备大型数据集的强大云端进行训练。因此，训练有效的深度学习模型已成为整个软件生命周期中的关键阶段。在训练深度学习模型（尤其是大模型）时，开发者需要在训练过程中将计算和内存资源并行化并分布到多个设备上，这称为分布式深度学习训练，简称分布式训练。然而，开发者在分布式训练过程中遇到的独特挑战尚未在软件工程社区中得到研究。鉴于当前基于深度学习的软件对分布式训练的依赖日益增强，本文旨在填补这一知识空白，并首次对开发者在分布式训练中遇到的问题进行全面研究。为此，我们分析了Stack Overflow和GitHub上报告的1131个关于使用这些框架的真实开发者问题。我们构建了一个包含30个类别的细粒度故障症状分类法，并总结了针对不同症状的常见修复模式。基于研究结果，我们提出了可操作的启示性研究方向，这些方向有望促进基于深度学习软件的分布式训练开发，例如：在设计测试或调试工具时关注频繁且常见的修复模式；开发针对通信配置的高效测试与调试技术以及网络配置分析的综合方法；设计新型多设备检查点回放技术以辅助复现；以及为云平台设计无服务器API。