This study highlights the potential of fine-tuned ChatGPT (GPT-3.5) for automatically scoring student written constructed responses using example assessment tasks in science education. Recent studies on OpenAI's generative model GPT-3.5 proved its superiority in predicting the natural language with high accuracy and human-like responses. GPT-3.5 has been trained over enormous online language materials such as journals and Wikipedia; therefore, more than direct usage of pre-trained GPT-3.5 is required for automatic scoring as students utilize a different language than trained material. These imply that a domain-specific model, fine-tuned over data for specific tasks, can enhance model performance. In this study, we fine-tuned GPT-3.5 on six assessment tasks with a diverse dataset of middle-school and high-school student responses and expert scoring. The six tasks comprise two multi-label and four multi-class assessment tasks. We compare the performance of fine-tuned GPT-3.5 with the fine-tuned state-of-the-art Google's generated language model, BERT. The results show that in-domain training corpora constructed from science questions and responses for BERT achieved average accuracy = 0.838, SD = 0.069. GPT-3.5 shows a remarkable average increase (9.1%) in automatic scoring accuracy (mean = 9.15, SD = 0.042) for the six tasks, p =0.001 < 0.05. Specifically, for multi-label tasks (item 1 with 5 labels; item 2 with 10 labels), GPT-3.5 achieved significantly higher scoring accuracy than BERT across all the labels, with the second item achieving a 7.1% increase. The average scoring increase for the four multi-class items for GPT-3.5 was 10.6% compared to BERT. Our study confirmed the effectiveness of fine-tuned GPT-3.5 for automatic scoring of student responses on domain-specific data in education with high accuracy. We have released fine-tuned models for public use and community engagement.

翻译：本研究凸显了微调后的ChatGPT（GPT-3.5）在科学教育情境下，利用示例评估任务对学生书面构建型回答进行自动评分的潜力。近期关于OpenAI生成式模型GPT-3.5的研究证明，该模型在自然语言预测方面具有高准确率及类人反应优势。由于GPT-3.5基于期刊、维基百科等海量网络语言材料训练，而学生使用语言与训练语料存在差异，因此直接应用预训练GPT-3.5无法满足自动评分需求。这表明，针对特定任务微调领域专用模型可提升模型性能。本研究基于六项评估任务，采用初中和高中学生多样化回答数据及专家评分对GPT-3.5进行微调。六项任务包含两项多标签分类任务与四项多分类任务。我们将微调后的GPT-3.5与基于谷歌生成式语言模型BERT的最先进微调方案进行对比。结果显示，基于科学问题与回答构建的领域内训练语料，BERT的平均准确率达0.838（标准差=0.069）。GPT-3.5在六项任务上的自动评分准确率平均提升9.1%（均值=9.15，标准差=0.042），p=0.001<0.05。具体而言，针对多标签任务（任务1含5个标签；任务2含10个标签），GPT-3.5在各标签上的评分准确率均显著高于BERT，其中第二个任务准确率提升7.1%。与BERT相比，GPT-3.5在四项多分类任务上的平均评分提升幅度为10.6%。本研究证实，微调后的GPT-3.5可在教育领域基于领域特定数据实现高准确率的学生回答自动评分。我们已将微调模型公开发布，供社区使用与参与。