The development of effective autograders is key for scaling assessment and feedback. While NLP based autograding systems for open-ended response questions have been found to be beneficial for providing immediate feedback, autograders are not always liked, understood, or trusted by students. Our research tested the effect of transparency on students' attitudes towards autograders. Transparent autograders increased students' perceptions of autograder accuracy and willingness to discuss autograders in survey comments, but did not improve other related attitudes -- such as willingness to be graded by them on a test -- relative to the control without transparency. However, this lack of impact may be due to higher measured student trust towards autograders in this study than in prior work in the field. We briefly discuss possible reasons for this trend.
翻译:开发有效的自动评分系统是扩大评估与反馈规模的关键。尽管基于自然语言处理的开放式问答题自动评分系统已被证实有助于提供即时反馈,但学生并不总是喜欢、理解或信任这类系统。本研究检验了透明度对学生自动评分系统态度的影响。透明化的自动评分系统提升了学生对评分准确性的感知,并增强了他们在调查评论中讨论该系统的意愿,但与未提供透明度的对照组相比,并未改善其他相关态度——例如在测试中接受其评分的意愿。然而,这种影响的缺失可能源于本研究中学生对自动评分系统的信任度高于该领域先前研究测得的结果。我们简要讨论了这一趋势的可能成因。