Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge from white-box generative LLMs is still under-explored, which becomes more and more important with the prosperity of LLMs. In this work, we propose MiniLLM that distills smaller language models from generative larger language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective optimization approach to learn this objective. Extensive experiments in the instruction-following setting show that the MiniLLM models generate more precise responses with the higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance. Our method is also scalable for different model families with 120M to 13B parameters. We will release our code and model checkpoints at https://aka.ms/MiniLLM.

翻译：知识蒸馏（KD）是一种有前景的技术，旨在降低大型语言模型（LLMs）的高计算需求。然而，以往的KD方法主要应用于白盒分类模型，或训练小模型模仿黑盒模型API（如ChatGPT）。如何有效蒸馏生成式白盒LLMs的知识仍是一个未被充分探索的问题，而随着LLMs的蓬勃发展，这一问题日益重要。在本工作中，我们提出了MiniLLM，用于从生成式大型语言模型中蒸馏出较小的语言模型。我们首先将标准KD方法中的前向Kullback-Leibler散度（KLD）目标替换为反向KLD，该目标更适合生成式语言模型的KD，以防止学生模型高估教师分布的低概率区域。随后，我们推导出一种有效的优化方法来学习这一目标。在指令遵循场景下的大量实验表明，MiniLLM模型生成的响应更精确，整体质量更高，曝光偏差更低，校准性更好，且长文本生成性能更优。我们的方法可扩展至不同模型系列，参数规模从1.2亿到130亿不等。我们将于https://aka.ms/MiniLLM发布代码及模型检查点。