Fine-tuned transformer models have shown superior performances in many natural language tasks. However, the large model size prohibits deploying high-performance transformer models on resource-constrained devices. This paper proposes a quantization-aware tensor-compressed training approach to reduce the model size, arithmetic operations, and ultimately runtime latency of transformer-based models. We compress the embedding and linear layers of transformers into small low-rank tensor cores, which significantly reduces model parameters. A quantization-aware training with learnable scale factors is used to further obtain low-precision representations of the tensor-compressed models. The developed approach can be used for both end-to-end training and distillation-based training. To improve the convergence, a layer-by-layer distillation is applied to distill a quantized and tensor-compressed student model from a pre-trained transformer. The performance is demonstrated in two natural language understanding tasks, showing up to $63\times$ compression ratio, little accuracy loss and remarkable inference and training speedup.

翻译：微调后的Transformer模型在许多自然语言任务中展现出优越性能。然而，庞大的模型尺寸阻碍了在资源受限设备上部署高性能Transformer模型。本文提出一种量化感知的张量压缩联合训练方法，旨在减小基于Transformer的模型尺寸、算术运算量并最终降低运行延迟。我们将Transformer的嵌入层和线性层压缩为小型低秩张量核心，显著减少模型参数。利用可学习缩放因子的量化感知训练，进一步获得张量压缩模型的低精度表示。该开发方法可同时用于端到端训练和基于蒸馏的训练。为改善收敛性，采用逐层蒸馏方法从预训练Transformer中蒸馏得到量化且张量压缩的学生模型。在两个自然语言理解任务上的性能验证表明，该方法可实现高达63倍的压缩比、极小的精度损失以及显著的推理与训练加速效果。