Low Rank Decomposition (LRD) is a model compression technique applied to the weight tensors of deep learning models in order to reduce the number of trainable parameters and computational complexity. However, due to high number of new layers added to the architecture after applying LRD, it may not lead to a high training/inference acceleration if the decomposition ranks are not small enough. The issue is that using small ranks increases the risk of significant accuracy drop after decomposition. In this paper, we propose two techniques for accelerating low rank decomposed models without requiring to use small ranks for decomposition. These methods include rank optimization and sequential freezing of decomposed layers. We perform experiments on both convolutional and transformer-based models. Experiments show that these techniques can improve the model throughput up to 60% during training and 37% during inference when combined together while preserving the accuracy close to that of the original models

翻译：低秩分解（LRD）是一种应用于深度学习模型权重张量的模型压缩技术，旨在减少可训练参数数量和计算复杂度。然而，由于应用LRD后网络架构新增了大量层，若分解秩不够小，则可能无法实现显著的训练/推理加速。问题在于，使用较小秩会增加分解后模型精度显著下降的风险。本文提出两种无需依赖小秩分解即可加速低秩分解模型的技术，包括秩优化与分解层的序列冻结方法。我们针对卷积神经网络和基于Transformer的模型进行了实验。结果表明，这两种技术联合使用时，可在保持与原始模型相近精度的前提下，将训练吞吐量提升高达60%，推理吞吐量提升高达37%。