Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters, the necessity to train overparametrized models remains poorly understood, and alternative approaches do not necessarily make it cheaper to train high-performance models. In this paper, we explore low-rank training techniques as an alternative approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to pre-training transformer language models with up to 350M parameters and demonstrate comparable performance to regular neural network training. Furthermore, we observe that the efficiency of ReLoRA increases with model size, making it a promising approach for training multi-billion-parameter networks efficiently. Our findings shed light on the potential of low-rank training techniques and their implications for scaling laws.

翻译：尽管规模扩展在拥有数千亿参数的大型网络中占据主导地位且效果显著，但训练过参数化模型的必要性仍未被充分理解，而替代方法也未必能降低高性能模型的训练成本。本文探索低秩训练技术作为训练大型神经网络的替代方案。我们提出一种名为ReLoRA的新方法，利用低秩更新来训练高秩网络。我们将ReLoRA应用于参数量高达3.5亿的Transformer语言模型预训练中，并证明其性能与常规神经网络训练相当。此外，我们观察到ReLoRA的效率随模型规模增大而提升，这使其成为高效训练数十亿参数网络的一种有前景的方法。我们的研究揭示了低秩训练技术的潜力及其对规模扩展定律的启示。