Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks. However, their computational costs are prohibitively high. To address this issue, previous research has attempted to distill the knowledge of LLMs into smaller models by generating annotated data. Nonetheless, these works have mainly focused on the direct use of LLMs for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. In this paper, we propose EvoKD: Evolving Knowledge Distillation, which leverages the concept of active learning to interactively enhance the process of data generation using large language models, simultaneously improving the task capabilities of small domain model (student model). Different from previous work, we actively analyze the student model's weaknesses, and then synthesize labeled samples based on the analysis. In addition, we provide iterative feedback to the LLMs regarding the student model's performance to continuously construct diversified and challenging samples. Experiments and analysis on different NLP tasks, namely, text classification and named entity recognition show the effectiveness of EvoKD.

翻译：大语言模型（LLMs）在各种自然语言处理任务中展现出卓越的能力。然而，其计算成本高昂。为解决该问题，先前研究尝试通过生成标注数据，将大语言模型的知识蒸馏至较小模型。然而，这些工作主要集中于直接利用大语言模型进行文本生成与标注，而未充分探索其理解目标任务并获取有价值知识的潜力。本文提出EvoKD：进化式知识蒸馏，该方法利用主动学习理念，交互式地增强基于大语言模型的数据生成过程，同时提升小型领域模型（学生模型）的任务能力。不同于先前工作，我们主动分析学生模型的薄弱环节，并基于该分析合成标注样本。此外，我们向大语言模型提供关于学生模型性能的迭代反馈，以持续构建多样化且具有挑战性的样本。针对文本分类与命名实体识别等不同自然语言处理任务的实验与分析证明了EvoKD的有效性。