Deep learning based recommendation models (DLRM) are widely used in several business critical applications. Training such recommendation models efficiently is challenging because they contain billions of embedding-based parameters, leading to significant overheads from embedding access. By profiling existing systems for DLRM training, we observe that around 75\% of the iteration time is spent on embedding access and model synchronization. Our key insight in this paper is that embedding access has a specific structure which can be used to accelerate training. We observe that embedding accesses are heavily skewed, with around 1\% of embeddings representing more than 92\% of total accesses. Further, we observe that during offline training we can lookahead at future batches to determine exactly which embeddings will be needed at what iteration in the future. Based on these insights, we develop Bagpipe, a system for training deep recommendation models that uses caching and prefetching to overlap remote embedding accesses with the computation. We design an Oracle Cacher, a new component that uses a lookahead algorithm to generate optimal cache update decisions while providing strong consistency guarantees against staleness. We also design a logically replicated, physically partitioned cache and show that our design can reduce synchronization overheads in a distributed setting. Finally, we propose a disaggregated system architecture and show that our design can enable low-overhead fault tolerance. Our experiments using three datasets and four models show that Bagpipe provides a speed up of up to 5.6x compared to state of the art baselines, while providing the same convergence and reproducibility guarantees as synchronous training.

翻译：基于深度学习的推荐模型（DLRM）广泛应用于多个关键业务应用中。高效训练此类推荐模型极具挑战性，因为模型包含数十亿基于嵌入的参数，导致嵌入访问产生显著开销。通过对现有DLRM训练系统的分析，我们观察到约75%的迭代时间耗费在嵌入访问与模型同步上。本文的核心洞察在于，嵌入访问具有特定结构，可据此加速训练。我们观察到嵌入访问呈现严重偏斜性：约1%的嵌入占据了总访问量的92%以上。此外，在离线训练过程中，我们可以通过预判未来批次，精确确定哪些嵌入将在未来哪个迭代中被需要。基于这些发现，我们开发了Bagpipe——一个通过缓存与预取技术将远程嵌入访问与计算重叠的深度推荐模型训练系统。我们设计了Oracle Cacher组件，它利用预判算法生成最优缓存更新决策，同时提供强一致性保证以防止数据陈旧。我们还设计了逻辑复制、物理分区的缓存结构，证明该设计能降低分布式环境中的同步开销。最终，我们提出一种解耦的系统架构，并证明该设计可实现低开销的容错机制。基于三个数据集和四个模型的实验表明，与现有最优基线相比，Bagpipe在提供与同步训练相同的收敛性与可重现性保证的同时，实现了最高5.6倍的加速比。