Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in \url{https://github.com/microsoft/LMOps/tree/main/minillm}.

翻译：知识蒸馏（KD）是一种有前景的技术，旨在降低大型语言模型（LLM）的高计算需求。然而，先前的KD方法主要应用于白盒分类模型，或训练小模型以模仿黑盒模型API（如ChatGPT）。如何有效将白盒LLM的知识蒸馏至小模型仍待充分探索，而随着开源LLM的蓬勃发展，这一问题变得愈发重要。本文提出一种将LLM蒸馏为较小语言模型的KD方法。我们首先将标准KD方法中的前向KL散度目标替换为反向KLD，前者更适用于生成语言模型的KD，可防止学生模型高估教师分布的低概率区域。随后，我们推导出一种有效的优化方法来学习该目标。所得学生模型被称为MiniLLM。在指令遵循场景下的大量实验表明，MiniLLM生成的响应更精准、整体质量更高、曝光偏差更低、校准性更优且长文本生成性能优于基线方法。本方法可扩展至参数规模从1.2亿到130亿的不同模型系列。我们的代码、数据和模型检查点可在\url{https://github.com/microsoft/LMOps/tree/main/minillm}获取。