Fine-tuned transformer models have shown superior performances in many natural language tasks. However, the large model size prohibits deploying high-performance transformer models on resource-constrained devices. This paper proposes a quantization-aware tensor-compressed training approach to reduce the model size, arithmetic operations, and ultimately runtime latency of transformer-based models. We compress the embedding and linear layers of transformers into small low-rank tensor cores, which significantly reduces model parameters. A quantization-aware training with learnable scale factors is used to further obtain low-precision representations of the tensor-compressed models. The developed approach can be used for both end-to-end training and distillation-based training. To improve the convergence, a layer-by-layer distillation is applied to distill a quantized and tensor-compressed student model from a pre-trained transformer. The performance is demonstrated in two natural language understanding tasks, showing up to $63\times$ compression ratio, little accuracy loss and remarkable inference and training speedup.

翻译：微调后的Transformer模型在多项自然语言任务中展现出优越性能，然而，庞大的模型规模阻碍了高性能Transformer模型在资源受限设备上的部署。本文提出一种量化感知的张量压缩训练方法，用于减少基于Transformer模型的模型规模、算术运算量以及最终运行延迟。我们将Transformer的嵌入层和线性层压缩为小型低秩张量核，显著减少了模型参数。通过引入具有可学习缩放因子的量化感知训练，进一步获取张量压缩模型的低精度表示。所开发的方法既可用于端到端训练，也可用于基于蒸馏的训练。为改善收敛性，采用逐层蒸馏方法，将预训练Transformer的知识蒸馏至量化且张量压缩的学生模型。在两个自然语言理解任务中的性能评估表明，该方法可实现高达63倍的压缩比，且几乎无精度损失，并显著提升推理与训练速度。