Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive. We study Matrix Product Operator (MPO) decomposition as a principled compression method for transformers. MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi. We replace every nn.Linear layer in PicoGPT, a GPT-2-style character-level language model with about 1M parameters, with an MPOLinear module parameterised as an MPO chain. Cores are initialised either by TT-SVD from pretrained dense weights or from random initialisation, and trained using standard PyTorch autograd without a custom backward pass. We derive balanced factorisation schemes for the five distinct weight shapes in PicoGPT and evaluate bond dimensions chi in {4, 8, 16, 32} on Tiny Shakespeare. MPO compression achieves up to 13x compression per transformer block at chi = 4. At chi = 16, the model uses 191,872 parameters instead of 1,020,224 while retaining 97.7% of baseline token accuracy (51.6% vs 52.8%). Reconstruction error follows the expected trend and is lower for three-site than two-site factorisations at the same bond dimension. The chi = 8 model gives the best accuracy per parameter, exceeding the dense baseline by 2.7x on this metric. These results show that MPO parameterisation is a practical and theoretically grounded alternative to low-rank methods and unstructured pruning for transformer compression.

翻译：基于Transformer的语言模型在各类自然语言处理任务中展现出强劲性能，但其参数量随隐藏维度呈二次方增长，导致在资源受限硬件上的部署成本高昂。本文研究将矩阵乘积算子（MPO）分解作为一种有理论依据的Transformer压缩方法。MPO将权重矩阵分解为低秩核心序列，通过键维度（bond dimension）χ控制逼近精度。我们以参数量约1M的GPT-2风格字符级语言模型PicoGPT为例，将其所有nn.Linear层替换为以MPO链参数化的MPOLinear模块。核心初始化可采用预训练稠密权重的TT-SVD分解或随机初始化法，并使用标准PyTorch自动求导机制进行训练，无需自定义反向传播。针对PicoGPT中五种不同权重形状推导出平衡分解方案，并在Tiny Shakespeare数据集上评估了χ∈{4,8,16,32}的键维度。当χ=4时，MPO压缩可使每个Transformer块获得13倍压缩比。当χ=16时，模型使用191,872个参数（基准模型为1,020,224个）仍保持基准模型97.7%的字符精度（51.6% vs 52.8%）。重构误差遵循预期趋势，在相同键维度下三位置分解优于两位置分解。χ=8模型取得了最佳单位参数精度，该指标比稠密基准模型提升2.7倍。实验表明，MPO参数化是Transformer压缩领域低秩方法与无结构剪枝的一种兼具实用性与理论基础的替代方案。