Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https://github.com/microsoft/LMOps/tree/main/minillm.

翻译：知识蒸馏（KD）是一种降低大型语言模型（LLMs）高计算需求的有效技术。然而，以往的KD方法主要应用于白盒分类模型，或训练小模型模仿ChatGPT等黑盒模型API。如何有效蒸馏白盒LLMs的知识至小模型仍待深入探索，而随着开源LLMs的蓬勃发展，这一问题变得愈发重要。本文提出一种将LLMs蒸馏至更小语言模型的KD方法。我们首先将标准KD方法中的前向KL散度（KLD）目标替换为反向KLD，该目标更适用于生成式语言模型的蒸馏，可防止学生模型高估教师分布的低概率区域。随后，我们推导出有效的优化方法来学习该目标。学生模型被命名为MiniLLM。在指令遵循设定下的广泛实验表明，MiniLLM相比基线方法能生成更精准的回答，并具有更高的整体质量、更低的曝光偏差、更优的校准性能及更强的长文本生成能力。本方法适用于参数规模从1.2亿至130亿的不同模型族。我们的代码、数据和模型检查点可在https://github.com/microsoft/LMOps/tree/main/minillm获取。