We study semi-supervised sequence generation tasks where labeled data are too scarce to effectively finetune a model and at the same time few-shot prompting of a large language model (LLM) has suboptimal performance. This happens when a task, such as parsing, is expensive to annotate and also unfamiliar to a pretrained LLM. In this paper, we present a discovery that student models distilled from an in-context learned LLM can often generalize better than their teacher on such tasks. Leveraging this finding, we present a new method -- multistage collaborative knowledge distillation from an LLM (MCKD) -- for such tasks. MCKD first few-shot prompts an LLM to produce pseudolabels for unlabeled data. At each intermediate knowledge distillation (KD) stage, a new pair of students is trained on disjoint partitions of the pseudolabeled data. Each student then produces new and improved pseudolabels for its unseen partition to be used in the next stage of distillation. We demonstrate the advantage of multistage cross-partition labeling on several syntactic and semantic parsing tasks. On CRAFT biomedical parsing, for example, 3-stage MCKD with 50 labeled examples outperforms the prompted LLM and vanilla KD by 7.5% and 3.7% parsing F1, respectively, and matches the performance of supervised finetuning with 500 examples.

翻译：我们研究半监督序列生成任务，其中标注数据过于稀少，无法有效微调模型，同时大语言模型（LLM）的少样本提示性能次优。这种情况发生在诸如解析等任务中——标注成本高昂，且预训练LLM对其不熟悉。本文发现，从上下文学习的LLM中蒸馏得到的学生模型，在此类任务上的泛化能力往往优于其教师模型。基于这一发现，我们提出了一种新方法——基于LLM的多阶段协作知识蒸馏（MCKD）。MCKD首先通过少样本提示LLM，为未标注数据生成伪标签。在每个中间知识蒸馏（KD）阶段，在伪标签数据的分离子集上训练一对新的学生模型，然后每个学生模型为其未见过的子集生成改进后的新伪标签，用于下一阶段蒸馏。我们在多个句法和语义解析任务上验证了多阶段跨分区标注的优势。例如，在CRAFT生物医学解析任务中，使用50个标注示例的三阶段MCKD，其解析F1值分别比提示LLM和普通KD高出7.5%和3.7%，并匹配了使用500个示例进行监督微调的性能。