Scaling DNNs is shown to deliver dramatic quality gains across ML problems. This, however, has also led to a concomitant quadratic increase in computation cost. To tackle this, along with the failure of accelerator memory capacity to keep up, training these models increasingly relies on distributed training techniques. As such, an important question of interest is: how will compute and communication relatively scale as models scale and hardware evolves? A careful study which answers this question can better guide the design of future systems. To this end, this work provides a comprehensive multi-axial (algorithmic, empirical, hardware evolution) analysis of compute vs. communication ($\textbf{Comp-vs.-Comm}$) scaling for future Transformer models on future hardware. Using algorithmic analysis we show that compute generally enjoys an edge over communication as models scale. However, when viewed through the lens of slower memory capacity scaling, these trends are being stressed. Next, we craft an empirical strategy to study Comp-vs.-Comm scaling for future models/hardware using existing hardware. This allows hundreds of future models/hardware scenarios to be studied at three orders of magnitude lower profiling costs. Our experiments demonstrate that communication will be a significant portion (about 40-75%) of execution as models and hardware evolve, and communication which is today hidden by overlapped computation will likely get exposed. Further, the generality of our strategy makes it a strong basis to perform Comp-vs.-Comm scaling analysis for any future model. Overall, this work underscores the increasingly large role communication will play as models scale.

翻译：研究表明，深度神经网络（DNN）的规模扩展能在各类机器学习任务中带来显著的质量提升，但这也导致计算成本呈二次方增长。为应对这一问题，并克服加速器内存容量无法同步扩展的瓶颈，分布式训练技术愈发成为训练大模型的关键手段。由此引出一个重要问题：随着模型规模扩大和硬件演进，计算与通信的相对扩展趋势将如何演变？对此问题的系统研究能为未来系统设计提供重要指导。为此，本文从算法分析、实验验证和硬件演进三个维度，对面向未来硬件上Transformer模型的计算与通信扩展（$\textbf{Comp-vs.-Comm}$）进行了全面分析。通过算法分析，我们发现在模型规模扩展时计算通常比通信更具优势。然而，从内存容量扩展相对滞后的视角来看，这一趋势正面临挑战。随后，我们设计了一套基于现有硬件研究未来模型/硬件场景下计算与通信扩展关系的实验策略，能以三个数量级更低的剖析成本研究数百种未来模型/硬件组合场景。实验结果表明，随着模型和硬件演进，通信将占据执行的显著部分（约40-75%），且目前通常被计算重叠所掩盖的通信行为很可能会暴露出来。此外，该策略的通用性使其能作为分析任何未来模型的计算与通信扩展关系的强有力基础。总体而言，本研究强调了通信在大规模模型扩展中将扮演日益重要的角色。