Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.

翻译：现代工业深度学习推荐模型通常通过分析序列交互历史来提取用户偏好，随后基于这些推导出的兴趣生成预测。数据特征固有的异质性常导致大规模训练期间计算资源严重利用率不足，这主要是由严重掉队节点和阻塞通信引发的计算气泡所致。本文提出FreeScale解决方案，旨在：(1) 通过精心负载均衡的输入样本缓解掉队节点问题；(2) 通过将优先级嵌入通信与计算重叠来最小化阻塞通信；(3) 利用无SM干扰通信技术，在计算与通信重叠过程中解决GPU资源竞争问题。实验评估表明，在256块H100 GPU上运行真实业务负载时，FreeScale最多可实现90.3%的计算气泡缩减。