Modern recommendation models have increased to trillions of parameters. As cluster scales expand to O(1k), distributed training bottlenecks shift from computation and memory to data movement, especially lookup and communication latency associated with embeddings. Existing solutions either optimize only one bottleneck or improve throughput by sacrificing training consistency. This paper presents NestPipe, a large-scale decentralized embedding training framework that tackles both bottlenecks while preserving synchronous training semantics. NestPipe exploits two hierarchical sparse parallelism opportunities through nested pipelining. At the inter-batch level, Dual-Buffer Pipelining (DBP) constructs a staleness-free five-stage pipeline through dual-buffer synchronization, mitigating lookup bottlenecks without embedding staleness. At the intra-batch level, we identify the embedding freezing phenomenon, which inspires Frozen-Window Pipelining (FWP) to overlap All2All communication with dense computation via coordinated stream scheduling and key-centric sample clustering. Experiments on production GPU and NPU clusters with 1,536 workers demonstrate that NestPipe achieves up to 3.06x speedup and 94.07% scaling efficiency.

翻译：摘要：现代推荐模型的参数量已增至万亿级别。随着集群规模扩展至千量级，分布式训练的瓶颈从计算与内存转向数据移动，特别是与嵌入层相关的查找和通信延迟。现有方案要么仅优化单一瓶颈，要么以牺牲训练一致性为代价提升吞吐量。本文提出NestPipe——一种大规模去中心化嵌入训练框架，在保持同步训练语义的同时解决上述双重瓶颈。NestPipe通过嵌套流水线技术利用两种层次化的稀疏并行性机会。在批次间层面，双缓冲流水线（DBP）通过双缓冲同步构建无停滞的五阶段流水线，在不引入嵌入陈旧性的前提下缓解查找瓶颈。在批次内层面，我们识别出嵌入冻结现象，受此启发提出冻结窗口流水线（FWP），通过协调流调度和以关键样本为中心的聚类方法，将全连接通信与稠密计算重叠。在包含1536个工作节点的生产级GPU与NPU集群上的实验表明，NestPipe可实现最高3.06倍加速比和94.07%的扩展效率。