In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: \textit{algorithm}, \textit{skill}, and \textit{verticalization} -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs.

翻译：在大语言模型（LLMs）时代，知识蒸馏（KD）作为一项关键技术，将GPT-4等领先商业LLM的先进能力迁移至LLaMA和Mistral等开源模型。同时，随着开源LLM的蓬勃发展，KD在模型压缩以及利用自身作为教师实现自我优化方面发挥着关键作用。本文全面综述了KD在LLM领域中的角色，重点阐述了其在向小型模型传递高级知识、模型压缩及自我优化中的关键功能。本综述严谨地围绕三个基础支柱构建：算法、技能与垂直领域——系统性地审视了KD机制、特定认知能力的增强及其在各领域的实际应用。尤为关键的是，综述探讨了数据增强（DA）与KD之间复杂的相互作用，揭示了DA如何作为KD框架内的强大范式提升LLM性能。通过利用DA生成上下文丰富、技能特定的训练数据，KD突破了传统边界，使开源模型能够逼近其商业对应模型所具备的上下文理解能力、伦理对齐特性及深层语义洞察力。本研究旨在为研究人员和实践者提供一份具有洞察力的指南，详细概述当前KD方法，并提出未来研究方向。重要的是，我们坚定倡导遵守规范LLM使用的法律条款，确保KD of LLMs的伦理与合法应用。相关Github仓库地址为：https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs。