This study proposes a method for distilling the knowledge of fine-tuned Large Language Models (LLMs) into a smaller, more efficient, and accurate neural network, specifically targeting the challenge of deploying these models on resource-constrained devices. Our methodology involves training the smaller student model using the prediction probabilities of the LLM, which serves as a teacher model. This is achieved through a specialized loss function tailored to learn from the LLM's output probabilities, ensuring that the student model closely mimics the teacher's performance. To test this approach, we utilized a large dataset, 7T, containing 6,684 student-written responses to science questions and three other datasets with student-written responses. We also compared performance with original neural network (NN) models to validate the accuracy. Results have shown that the NN and distilled student models have comparable accuracy to the teacher model for the 7T dataset; however, other datasets have shown significantly lower accuracy (28% on average) for NN, though our proposed distilled model is still able to achieve 12\% higher accuracy than NN. Furthermore, the student model size ranges from 0.1M to 0.02M, 100 times smaller in terms of parameters and ten times smaller compared with the original output model size. The significance of this research lies in its potential to make advanced AI technologies accessible in typical educational settings, particularly for automatic scoring.

翻译：本研究提出一种方法，将微调后的大语言模型（LLM）知识蒸馏到更小、更高效且更精准的神经网络中，专门针对在资源受限设备上部署此类模型的挑战。我们的方法利用LLM（作为教师模型）的预测概率来训练较小的学生模型。通过设计一种专门从LLM输出概率中学习的损失函数，确保学生模型能紧密模仿教师模型的性能。为验证该方法，我们使用了包含6684份学生对科学问题作答的大型数据集7T，以及另外三个包含学生书面回答的数据集。同时，我们还与原始神经网络（NN）模型进行性能对比以验证准确性。结果表明，在7T数据集上，NN和蒸馏学生模型均能达到与教师模型相当的准确率；然而在其他数据集中，NN的准确率显著降低（平均降低28%），但本文提出的蒸馏模型仍比NN高出12%的准确率。此外，学生模型参数量介于0.1M至0.02M之间，其规模在参数量上比原始输出模型小100倍，在模型尺寸上小10倍。本研究的意义在于其潜在能力，能够使先进AI技术在典型教育场景（尤其是自动评分）中实现广泛应用。