As distributed machine learning (ML) workloads scale to thousands of GPUs connected by ultra-high-speed inter-connects, tail latency in collective communication has emerged as a primary bottleneck. Prior RDMA designs, like RoCE, IRN, and SRNIC, enforce strict reliability and in-order delivery, relying on retransmissions and packet sequencing to ensure correctness. While effective for general-purpose workloads, these mechanisms introduce complexity and latency that scale poorly, where even rare packet losses or delays can consistently degrade system performance. We introduce Celeris, a domain-specific RDMA transport that revisits traditional reliability guarantees based on ML's tolerance for lost or partial data. Celeris removes retransmissions and in-order delivery from the RDMA NIC, enabling best-effort transport that exploits the robustness of ML workloads. It retains congestion control (e.g., DCQCN) and manages communication with software-level mechanisms such as adaptive timeouts and data prioritization, while shifting loss recovery to the ML pipeline (e.g., using the Hadamard Transform). Early results show that Celeris reduces 99th-percentile latency by up to 2.3x, cuts BRAM usage by 67%, and nearly doubles NIC resilience to faults -- delivering a resilient, scalable transport tailored for ML at cluster scale.

翻译：随着分布式机器学习（ML）工作负载扩展到由超高速互连连接的数千个GPU，集体通信中的尾部延迟已成为主要瓶颈。现有的RDMA设计，如RoCE、IRN和SRNIC，强制实施严格的可靠性和顺序交付，依赖重传和数据包排序来确保正确性。虽然这些机制对通用工作负载有效，但它们引入了复杂性和延迟，且扩展性不佳——即使罕见的数据包丢失或延迟也会持续降低系统性能。我们提出了Celeris，一种面向特定领域的RDMA传输协议，它基于机器学习对丢失或部分数据的容忍性重新审视了传统的可靠性保证。Celeris从RDMA网卡中移除了重传和顺序交付机制，实现了利用ML工作负载鲁棒性的尽力而为传输。它保留了拥塞控制（如DCQCN），并通过自适应超时和数据优先级等软件级机制管理通信，同时将丢失恢复转移到ML流水线中（例如使用Hadamard变换）。初步结果表明，Celeris将第99百分位延迟降低了最多2.3倍，BRAM使用量减少了67%，并将网卡对故障的恢复能力提升了近一倍——为集群规模的ML提供了量身定制的、具备弹性且可扩展的传输方案。