Energy efficiency of training and inferencing with large neural network models is a critical challenge facing the future of sustainable large-scale machine learning workloads. This paper introduces an alternative strategy, called phantom parallelism, to minimize the net energy consumption of traditional tensor (model) parallelism, the most energy-inefficient component of large neural network training. The approach is presented in the context of feed-forward network architectures as a preliminary, but comprehensive, proof-of-principle study of the proposed methodology. We derive new forward and backward propagation operators for phantom parallelism, implement them as custom autograd operations within an end-to-end phantom parallel training pipeline and compare its parallel performance and energy-efficiency against those of conventional tensor parallel training pipelines. Formal analyses that predict lower bandwidth and FLOP counts are presented with supporting empirical results on up to 256 GPUs that corroborate these gains. Experiments are shown to deliver approximately 50% reduction in the energy consumed to train FFNs using the proposed phantom parallel approach when compared with conventional tensor parallel methods. Additionally, the proposed approach is shown to train smaller phantom models to the same model loss on smaller GPU counts as larger tensor parallel models on larger GPU counts offering the possibility for even greater energy savings.

翻译：大型神经网络模型在训练与推理过程中的能耗效率，是未来可持续大规模机器学习工作负载面临的关键挑战。本文提出一种称为幻象并行的替代策略，旨在最小化传统张量（模型）并行——这一大型神经网络训练中能耗效率最低的组成部分——的净能耗消耗。该方法以前馈网络架构为背景进行阐述，作为对所提方法论的初步但全面的原理验证研究。我们推导了适用于幻象并行的新型前向与反向传播算子，将其实现为端到端幻象并行训练流程中的自定义自动微分操作，并将其并行性能与能耗效率同传统张量并行训练流程进行对比。本文提出了预测更低带宽与浮点运算次数的理论分析，并辅以在多达256个GPU上获得的实证结果，验证了这些性能增益。实验表明，相较于传统张量并行方法，采用所提幻象并行方法训练前馈网络可降低约50%的能耗。此外，研究证明所提方法能够以更少的GPU数量训练较小的幻象模型达到与较多GPU数量下较大张量并行模型相同的损失值，从而为实现更大的节能潜力提供了可能。