Under gang scheduling for large-scale distributed large language model (LLM) training, a single network anomaly can stall or abort an entire job. Current network fault tolerance mechanisms typically adopt a ``fallback and bypass'' approach within the switching fabric and at the access layer, tolerating in-network and access-layer failures. We explore whether RDMA fault tolerance can be extended to the cross-NIC level by failing over traffic to intra-host backup NICs. For the first time, we prove a fundamental Trilemma: it is impossible to have Cross-NIC RDMA failover that simultaneously preserves Exactly-Once Execution, Receiver-NIC Opacity, and a Zero-Copy datapath. Fortunately, we observe that dominant training frameworks (e.g., NCCL) rely on idempotent bulk transfers that tolerate relaxed memory ordering, as long as notification ordering is preserved. Leveraging this insight, we present SHIFT, a user-space RDMA layer that provides cross-NIC fault tolerance while preserving correct memory semantics. We implement SHIFT in \texttt{rdma-core} and evaluate it with PyTorch distributed training. Results show that SHIFT incurs negligible overhead during normal operation and successfully masks fatal NIC failures and link anomalies, allowing training to continue without costly restarts.

翻译：在大规模分布式大语言模型训练的组调度机制下，单个网络异常可能导致整个作业停滞或中止。当前的网络容错机制通常在交换结构和接入层采用“回退与旁路”方法，以容忍网络内部及接入层故障。本研究探讨了是否可通过将流量故障切换至主机内备份网卡，将RDMA容错能力扩展至跨网卡级别。我们首次证明了一个根本性的三难困境：不可能同时实现保持精确一次执行、接收方网卡操作透明性以及零拷贝数据路径的跨网卡RDMA故障切换。幸运的是，我们观察到主流训练框架（如NCCL）依赖于幂等批量传输，这些传输可容忍宽松的内存排序，只要通知顺序得以保持。基于这一发现，我们提出了SHIFT——一个用户空间的RDMA层，它在提供跨网卡容错能力的同时保持正确的内存语义。我们在\texttt{rdma-core}中实现了SHIFT，并通过PyTorch分布式训练进行评估。结果表明，SHIFT在正常运行期间产生的开销可忽略不计，并能成功屏蔽致命的网卡故障与链路异常，使得训练无需代价高昂的重启即可持续进行。