Machine learning-based automatic scoring can be challenging if students' responses are unbalanced across scoring categories, as it introduces uncertainty in the machine training process. To meet this challenge, we introduce a novel text data augmentation framework leveraging GPT-4, a generative large language model, specifically tailored for unbalanced datasets in automatic scoring. Our experimental dataset comprised student written responses to two science items. We crafted prompts for GPT-4 to generate responses resembling student written answers, particularly for the minority scoring classes, to augment the data. We then finetuned DistillBERT for automatic scoring based on the augmented and original datasets. Model performance was assessed using accuracy, precision, recall, and F1 metrics. Our findings revealed that incorporating GPT-4-augmented data remarkedly improved model performance, particularly for precision, recall, and F1 scores. Interestingly, the extent of improvement varied depending on the specific dataset and the proportion of augmented data used. Notably, we found that a varying amount of augmented data (5\%-40\%) was needed to obtain stable improvement for automatic scoring. We also compared the accuracies of models trained with GPT-4 augmented data to those trained with additional student-written responses. Results suggest that the GPT-4 augmented scoring models outperform or match the models trained with student-written augmented data. This research underscores the potential and effectiveness of data augmentation techniques utilizing generative large language models--GPT-4 in addressing unbalanced datasets within automated assessment.

翻译：基于机器学习的自动评分在学生回答数据按评分类别分布不平衡时面临挑战，这种不平衡性会引入训练过程的不确定性。为此，我们提出了一种新颖的文本数据增强框架，该框架利用生成式大语言模型GPT-4，专门针对自动评分中的不平衡数据集进行优化。实验数据包含学生对两个科学问题的书面回答。我们设计提示词引导GPT-4生成类似学生书面回答的文本，特别是为少数评分类别补充数据，从而增强原始数据集。随后基于增强后的数据集和原始数据集，微调DistillBERT进行自动评分。采用准确率、精确率、召回率和F1值评估模型性能。研究发现，融入GPT-4增强数据显著提升了模型性能，尤其体现在精确率、召回率和F1分数上。值得注意的是，性能提升幅度取决于具体数据集和增强数据的比例。实验表明，为稳定提升自动评分效果，需使用5%-40%不同比例的增强数据。此外，我们将基于GPT-4增强数据训练的模型与基于额外学生书面回答训练的模型进行精度对比，结果显示GPT-4增强的评分模型性能优于或持平于人工数据增强的模型。本研究揭示了利用生成式大语言模型GPT-4进行数据增强在解决自动化评估中数据集不平衡问题方面的潜力与有效性。