The grading of open-ended questions is a high-effort, high-impact task in education. Automating this task promises a significant reduction in workload for education professionals, as well as more consistent grading outcomes for students, by circumventing human subjectivity and error. While recent breakthroughs in AI technology might facilitate such automation, this has not been demonstrated at scale. It this paper, we introduce a novel automatic short answer grading (ASAG) system. The system is based on a fine-tuned open-source transformer model which we trained on large set of exam data from university courses across a large range of disciplines. We evaluated the trained model's performance against held-out test data in a first experiment and found high accuracy levels across a broad spectrum of unseen questions, even in unseen courses. We further compared the performance of our model with that of certified human domain experts in a second experiment: we first assembled another test dataset from real historical exams - the historic grades contained in that data were awarded to students in a regulated, legally binding examination process; we therefore considered them as ground truth for our experiment. We then asked certified human domain experts and our model to grade the historic student answers again without disclosing the historic grades. Finally, we compared the hence obtained grades with the historic grades (our ground truth). We found that for the courses examined, the model deviated less from the official historic grades than the human re-graders - the model's median absolute error was 44 % smaller than the human re-graders', implying that the model is more consistent than humans in grading. These results suggest that leveraging AI enhanced grading can reduce human subjectivity, improve consistency and thus ultimately increase fairness.

翻译：开放式问题的评分是教育领域一项高投入、高影响的任务。通过规避人类主观性和错误，自动化这一任务有望显著减轻教育工作者的工作负担，并为学生带来更一致的评分结果。尽管近期AI技术的突破可能促进此类自动化，但这尚未在大规模范围内得到验证。本文介绍了一种新型的自动简答题评分（ASAG）系统。该系统基于一个经过微调的开源Transformer模型，我们在涵盖多学科的大量大学课程考试数据上对其进行了训练。在第一项实验中，我们针对保留的测试数据评估了训练模型的性能，发现其在广泛未见问题（甚至未见课程）上均达到了高准确率。在第二项实验中，我们进一步将模型性能与经过认证的人类领域专家进行了比较：首先，我们从真实历史考试中构建了另一个测试数据集——这些数据中的历史评分是在受监管、具有法律约束力的考试过程中授予学生的；因此，我们将其视为实验的基准真值。随后，我们要求经过认证的人类领域专家和我们的模型在不公开历史评分的情况下，重新对历史学生答案进行评分。最后，我们将由此获得的评分与历史评分（基准真值）进行比较。结果发现，在受检课程中，模型偏离官方历史评分的程度小于人类重新评分者——模型的中位绝对误差比人类重新评分者小44%，这表明模型在评分方面比人类更具一致性。这些结果表明，利用AI增强评分可以减少人类主观性，提高一致性，并最终提升公平性。