Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training.

翻译：同步预训练任务在极大规模运行时，其性能受尾延迟支配。我们提出一种三管齐下的方法：（1）基于RDMA的新型传输协议MRC，该协议在多个路径间进行数据喷洒并主动实现负载均衡，消除流冲突问题；（2）采用多平面Clos拓扑结构以获取高交换机端口密度与冗余性的优势，使得超过10万GPU的训练集群可通过两层拓扑构建，同时增强物理冗余；（3）利用基于SRv6的静态源路由机制，赋予MRC自主绕过故障的自由。我们分享了在OpenAI与微软最大规模训练集群中将MRC与静态SRv6路由投入生产的实践经验——该方案已成功用于最新前沿模型的训练。实验证明，MRC使AI训练任务能够在过去会导致训练中断的多数网络故障中持续运行。