Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditional distributed systems or datacenter networks fall short and can hardly apply to real-world training systems. In this paper, we present EROICA, the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems (or the mixture of both). EROICA effectively summarizes runtime behavior patterns of LMT function executions via online profiling, and leverages differential observability to localize the root cause with minimal production impact. EROICA has been deployed as a production service for large-scale GPU clusters of ~100,000 GPUs for 1.5 years. It has diagnosed a variety of difficult performance issues with 97.5% success.

翻译：大规模模型训练的性能故障排查极具挑战性，这源于现代GPU集群前所未有的规模、软硬件交互的复杂性以及训练过程的数据密集性。为传统分布式系统或数据中心网络设计的现有故障排查方法存在不足，难以应用于实际训练系统。本文提出EROICA，首个在线故障排查系统，它基于性能剖析提供细粒度观测，并覆盖GPU集群中的所有机器，以诊断生产环境中的性能问题，包括硬件和软件问题（或两者混合）。EROICA通过在线剖析有效总结大规模模型训练函数执行的运行时行为模式，并利用差分可观测性以最小化生产影响来定位根本原因。EROICA已作为生产服务部署于约10万个GPU的大规模集群中达1.5年，成功诊断了多种复杂性能问题，成功率高达97.5%。