DLRM is a state-of-the-art recommendation system model that has gained widespread adoption across various industry applications. The large size of DLRM models, however, necessitates the use of multiple devices/GPUs for efficient training. A significant bottleneck in this process is the time-consuming all-to-all communication required to collect embedding data from all devices. To mitigate this, we introduce a method that employs error-bounded lossy compression to reduce the communication data size and accelerate DLRM training. We develop a novel error-bounded lossy compression algorithm, informed by an in-depth analysis of embedding data features, to achieve high compression ratios. Moreover, we introduce a dual-level adaptive strategy for error-bound adjustment, spanning both table-wise and iteration-wise aspects, to balance the compression benefits with the potential impacts on accuracy. We further optimize our compressor for PyTorch tensors on GPUs, minimizing compression overhead. Evaluation shows that our method achieves a 1.38$\times$ training speedup with a minimal accuracy impact.

翻译：DLRM是一种先进的推荐系统模型，已在各行业应用中广泛采用。然而，DLRM模型的庞大规模需要使用多设备/GPU进行高效训练。该过程中的一个主要瓶颈是从所有设备收集嵌入数据所需的耗时全对全通信。为缓解此问题，我们提出一种采用误差有界有损压缩的方法，以减少通信数据量并加速DLRM训练。我们基于对嵌入数据特征的深入分析，开发了一种新颖的误差有界有损压缩算法，以实现高压缩比。此外，我们引入了一种双级自适应策略用于误差边界调整，涵盖表级和迭代级两个层面，以平衡压缩收益与对准确性的潜在影响。我们进一步针对GPU上的PyTorch张量优化了压缩器，以最小化压缩开销。评估表明，我们的方法在实现1.38$\times$训练加速的同时，对准确性的影响微乎其微。