Low-rank compression, a popular model compression technique that produces compact convolutional neural networks (CNNs) with low rankness, has been well-studied in the literature. On the other hand, low-rank training, as an alternative way to train low-rank CNNs from scratch, has been exploited little yet. Unlike low-rank compression, low-rank training does not need pre-trained full-rank models, and the entire training phase is always performed on the low-rank structure, bringing attractive benefits for practical applications. However, the existing low-rank training solutions still face several challenges, such as a considerable accuracy drop and/or still needing to update full-size models during the training. In this paper, we perform a systematic investigation on low-rank CNN training. By identifying the proper low-rank format and performance-improving strategy, we propose ELRT, an efficient low-rank training solution for high-accuracy, high-compactness, low-rank CNN models. Our extensive evaluation results for training various CNNs on different datasets demonstrate the effectiveness of ELRT.

翻译：低秩压缩作为一种通过引入低秩结构生成紧凑卷积神经网络（CNN）的主流模型压缩技术，已在文献中得到广泛研究。然而，作为从零开始训练低秩CNN的替代方案，低秩训练方法至今尚未得到充分探索。与低秩压缩不同，低秩训练无需预训练的全秩模型，且整个训练阶段始终在低秩结构上进行，为实际应用带来了显著优势。然而，现有低秩训练方案仍面临诸多挑战，例如显著的精度下降和/或训练过程中仍需更新全尺寸模型。本文对低秩CNN训练进行了系统性研究，通过确定合适的低秩格式和性能提升策略，提出了ELRT——一种面向高精度、高紧凑性低秩CNN模型的高效低秩训练方案。我们在不同数据集上对多种CNN进行训练的广泛评估结果充分验证了ELRT的有效性。