Recent research has shown that training low-rank neural networks can effectively reduce the total number of trainable parameters without sacrificing predictive accuracy, resulting in end-to-end speedups. However, low-rank model training necessitates adjusting several additional factorization hyperparameters, such as the rank of the factorization at each layer. In this paper, we tackle this challenge by introducing Cuttlefish, an automated low-rank training approach that eliminates the need for tuning factorization hyperparameters. Cuttlefish leverages the observation that after a few epochs of full-rank training, the stable rank (i.e., an approximation of the true rank) of each layer stabilizes at a constant value. Cuttlefish switches from full-rank to low-rank training once the stable ranks of all layers have converged, setting the dimension of each factorization to its corresponding stable rank. Our results show that Cuttlefish generates models up to 5.6 times smaller than full-rank models, and attains up to a 1.2 times faster end-to-end training process while preserving comparable accuracy. Moreover, Cuttlefish outperforms state-of-the-art low-rank model training methods and other prominent baselines. The source code for our implementation can be found at: https://github.com/hwang595/Cuttlefish.

翻译：近期研究表明，训练低秩神经网络可在不牺牲预测精度的前提下有效减少可训练参数总量，从而实现端到端加速。然而，低秩模型训练需要调整多项额外的因子分解超参数（例如每层分解的秩）。本文通过提出Cuttlefish这一自动化低秩训练方法来解决该挑战，该方法消除了对因子分解超参数调优的需求。Cuttlefish利用以下观察现象：经过若干轮次的全秩训练后，各层的稳定秩（即真实秩的近似值）会稳定在恒定值。当所有层的稳定秩收敛后，Cuttlefish即从全秩训练切换至低秩训练，并将各因子分解维度设置为对应的稳定秩。实验结果表明，Cuttlefish生成的模型比全秩模型小5.6倍，端到端训练速度提升达1.2倍，同时保持可比的精度。此外，Cuttlefish优于当前最先进的低秩模型训练方法及其他重要基线方法。实现代码请参见：https://github.com/hwang595/Cuttlefish。