Matrix and tensor-guided parametrization for Natural Language Processing (NLP) models is fundamentally useful for the improvement of the model's systematic efficiency. However, the internal links between these two algebra structures and language model parametrization are poorly understood. Also, the existing matrix and tensor research is math-heavy and far away from machine learning (ML) and NLP research concepts. These two issues result in the recent progress on matrices and tensors for model parametrization being more like a loose collection of separate components from matrix/tensor and NLP studies, rather than a well-structured unified approach, further hindering algorithm design. To this end, we propose a unified taxonomy, which bridges the matrix/tensor compression approaches and model compression concepts in ML and NLP research. Namely, we adopt an elementary concept in linear algebra, that of a subspace, which is also the core concept in geometric algebra, to reformulate the matrix/tensor and ML/NLP concepts (e.g. attention mechanism) under one umbrella. In this way, based on our subspace formalization, typical matrix and tensor decomposition algorithms can be interpreted as geometric transformations. Finally, we revisit recent literature on matrix- or tensor-guided language model compression, rephrase and compare their core ideas, and then point out the current research gap and potential solutions.

翻译：矩阵与张量引导的参数化方法对于提升自然语言处理（NLP）模型的系统效率具有基础性意义。然而，这两种代数结构与语言模型参数化之间的内在联系尚未得到充分理解。此外，现有矩阵与张量研究数学性过强，与机器学习（ML）及NLP研究概念存在显著脱节。这两个问题导致当前基于矩阵和张量的模型参数化研究更像是矩阵/张量领域与NLP研究的松散组件集合，而非结构严谨的统一框架，进一步阻碍了算法设计。为此，我们提出一种统一分类法，旨在连接矩阵/张量压缩方法与ML/NLP研究中的模型压缩概念。具体而言，我们采用线性代数中的基本概念——子空间（这也是几何代数的核心概念），将矩阵/张量概念与ML/NLP概念（例如注意力机制）置于同一理论框架下重新表述。基于这种子空间形式化体系，典型的矩阵与张量分解算法可被阐释为几何变换。最后，我们系统回顾了近期基于矩阵或张量引导的语言模型压缩文献，对其核心思想进行重述与比较，进而指出当前研究空白与潜在解决方案。