Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning. However, the large model size poses challenges for practical deployment. To solve this problem, Quantization-Aware Training (QAT) has become increasingly popular. However, current QAT methods for generative models have resulted in a noticeable loss of accuracy. To counteract this issue, we propose a novel knowledge distillation method specifically designed for GLMs. Our method, called token-scaled logit distillation, prevents overfitting and provides superior learning from the teacher model and ground truth. This research marks the first evaluation of ternary weight quantization-aware training of large-scale GLMs with less than 1.0 degradation in perplexity and achieves enhanced accuracy in tasks like common-sense QA and arithmetic reasoning as well as natural language understanding. Our code is available at https://github.com/aiha-lab/TSLD.

翻译：生成式语言模型（GLMs）在文本生成、理解与推理等任务中展现出卓越性能。然而，模型规模庞大给实际部署带来了挑战。为解决该问题，量化感知训练（QAT）日益受到关注。但当前的生成式模型QAT方法导致显著的精度损失。针对此问题，我们提出了一种专为GLMs设计的新型知识蒸馏方法——token缩放的logit蒸馏。该方法能够防止过拟合，并从教师模型与真实标签中实现更优的学习。本研究首次对大规模GLMs的三值权重量化感知训练进行评估，在困惑度下降低于1.0的情况下，实现了常识问答、算术推理及自然语言理解等任务准确率的提升。我们的代码开源在 https://github.com/aiha-lab/TSLD。