Modern advancements in large-scale machine learning would be impossible without the paradigm of data-parallel distributed computing. Since distributed computing with large-scale models imparts excessive pressure on communication channels, significant recent research has been directed toward co-designing communication compression strategies and training algorithms with the goal of reducing communication costs. While pure data parallelism allows better data scaling, it suffers from poor model scaling properties. Indeed, compute nodes are severely limited by memory constraints, preventing further increases in model size. For this reason, the latest achievements in training giant neural network models also rely on some form of model parallelism. In this work, we take a closer theoretical look at Independent Subnetwork Training (IST), which is a recently proposed and highly effective technique for solving the aforementioned problems. We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication, and provide a precise analysis of its optimization performance on a quadratic model.

翻译：现代大规模机器学习的进步离不开数据并行分布式计算这一范式。由于使用大规模模型进行分布式计算会对通信信道施加过大压力，近期大量研究致力于协同设计通信压缩策略与训练算法，以期降低通信成本。纯数据并行虽能实现更好的数据扩展，但存在模型扩展属性较差的缺陷。具体而言，计算节点受内存约束严重限制，阻碍了模型规模的进一步扩大。因此，训练巨型神经网络模型的最新成果同样依赖于某种形式的模型并行。本文中，我们对近期提出的、用于解决上述问题的高效技术——独立子网训练（IST）——进行了更深入的理论审视。我们识别出IST与其他方法（如压缩通信的分布式方法）之间的本质差异，并在二次模型上对其优化性能进行了精确分析。