Large language models (LLMs) can acquire strong code-generation capabilities through few-shot learning. In contrast, supervised fine-tuning is still needed for smaller models to achieve good performance. Such fine-tuning demands a large number of task-specific NL-code pairs, which are expensive to obtain. In this paper, we attempt to transfer the code generation ability of an LLM to a smaller model with the aid of weakly-supervised data. More specifically, we propose explicit knowledge transfer (EKT), which uses the few-shot capabilities of a teacher LLM to create NL-code pairs that we then filter for correctness and fine-tune the student on. We evaluate EKT on the task of generating code solutions to math word problems from the GSM8k dataset. We find that EKT not only yields better performance than training with expert iteration, but also outperforms knowledge distillation, another form of knowledge transfer. A GPT-Neo 1.3B model trained using EKT with a GPT-J teacher achieves a 12.4% pass@100 on GSM8k, while the same student and teacher trained with knowledge distillation yield only a 3.7% pass@100. We also show that it is possible for a student model to outperform the teacher using EKT.

翻译：大语言模型（LLMs）可通过少样本学习获得强大的代码生成能力。然而，小型模型仍需通过监督式微调才能实现良好性能，这类微调需要大量任务特定的自然语言-代码对数据，获取成本高昂。本文尝试借助弱监督数据，将大语言模型的代码生成能力迁移至较小模型。具体而言，我们提出显式知识迁移（EKT）方法，利用教师大语言模型的少样本能力生成自然语言-代码对，随后筛选正确数据并对学生模型进行微调。我们在GSM8k数据集的数学应用题代码生成任务上评估EKT方法。实验表明，EKT不仅性能优于专家迭代训练，还超越了另一种知识迁移形式——知识蒸馏。采用EKT方法，以GPT-J为教师训练GPT-Neo 1.3B模型在GSM8k上取得12.4%的pass@100，而使用知识蒸馏训练相同学生模型与教师仅获得3.7%的pass@100。我们同时证明，学生模型可通过EKT方法超越教师模型性能。