In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: \textit{algorithm}, \textit{skill}, and \textit{verticalization} -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs.

翻译：在大语言模型时代，知识蒸馏成为将GPT-4等领先专有大语言模型的先进能力迁移至LLaMA、Mistral等开源模型的关键技术。此外，随着开源大语言模型的蓬勃发展，知识蒸馏在模型压缩以及通过将模型自身作为教师实施自我改进等方面发挥着重要作用。本文全面综述了知识蒸馏在大语言模型领域的作用，重点阐述了其在向小型模型传授先进知识方面的关键功能，以及在模型压缩和自我改进中的实用性。本综述精心构建了三大基础支柱：算法、技能和垂直化——系统性地剖析了知识蒸馏机制、特定认知能力的增强及其在多个领域的实际应用。尤为重要的是，本综述深入探讨了数据增强与知识蒸馏之间的复杂交互关系，阐明了数据增强如何作为知识蒸馏框架中的强大范式来提升大语言模型的性能。通过利用数据增强生成富含上下文且针对特定技能的训练数据，知识蒸馏突破了传统边界，使开源模型能够逼近其专有对手所具备的上下文理解能力、伦理对齐能力以及深层语义洞察能力。本研究旨在为研究人员和实践者提供一份富有洞见的指南，系统概述当前知识蒸馏方法论，并提出未来研究方向。我们坚决倡导遵守规范大语言模型使用的法律条款，确保知识蒸馏的应用符合伦理与法律要求。相关GitHub仓库地址为：https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs。