The components underpinning PLMs -- large weight matrices -- were shown to bear considerable redundancy. Matrix factorization, a well-established technique from matrix theory, has been utilized to reduce the number of parameters in PLM. However, it fails to retain satisfactory performance under moderate to high compression rate. In this paper, we identify the \textit{full-rankness} of fine-tuned PLM as the fundamental bottleneck for the failure of matrix factorization and explore the use of network pruning to extract low-rank sparsity pattern desirable to matrix factorization. We find such low-rank sparsity pattern exclusively exists in models generated by first-order pruning, which motivates us to unite the two approaches and achieve more effective model compression. We further propose two techniques: sparsity-aware SVD and mixed-rank fine-tuning, which improve the initialization and training of the compression procedure, respectively. Experiments on GLUE and question-answering tasks show that the proposed method has superior compression-performance trade-off compared to existing approaches.

翻译：构成预训练语言模型（PLM）底层组件的大权重矩阵已被证明存在显著冗余。矩阵分解作为矩阵理论中一项成熟技术，已被用于减少PLM中的参数量。然而，在中高压缩率下，该技术难以保持令人满意的性能。在本文中，我们识别出微调后PLM的“满秩性”是矩阵分解失效的根本瓶颈，并探索利用网络剪枝提取适合矩阵分解的低秩稀疏模式。我们发现这种低秩稀疏模式仅存在于由一阶剪枝生成的模型中，这促使我们将两种方法结合以实现更有效的模型压缩。我们进一步提出两种技术：稀疏感知奇异值分解（sparsity-aware SVD）与混合秩微调，它们分别改进了压缩过程的初始化与训练。在GLUE和问答任务上的实验表明，与现有方法相比，所提出方法在压缩与性能权衡方面具有更优表现。