Electronic exams (e-exams) have the potential to substantially reduce the effort required for conducting an exam through automation. Yet, care must be taken to sacrifice neither task complexity nor constructive alignment nor grading fairness in favor of automation. To advance automation in the design and fair grading of (functional programming) e-exams, we introduce the following: A novel algorithm to check Proof Puzzles based on finding correct sequences of proof lines that improves fairness compared to an existing, edit distance based algorithm; an open-source static analysis tool to check source code for task relevant features by traversing the abstract syntax tree; a higher-level language and open-source tool to specify regular expressions that makes creating complex regular expressions less error-prone. Our findings are embedded in a complete experience report on transforming a paper exam to an e-exam. We evaluated the resulting e-exam by analyzing the degree of automation in the grading process, asking students for their opinion, and critically reviewing our own experiences. Almost all tasks can be graded automatically at least in part (correct solutions can almost always be detected as such), the students agree that an e-exam is a fitting examination format for the course but are split on how well they can express their thoughts compared to a paper exam, and examiners enjoy a more time-efficient grading process while the point distribution in the exam results was almost exactly the same compared to a paper exam.
翻译:电子考试(e-exams)通过自动化手段,有望显著降低考试实施的工作量。然而,在追求自动化的同时,必须确保不牺牲任务复杂度、建设性对齐(constructive alignment)或评分公平性。为推进(函数式编程)电子考试设计与公平评分的自动化,本文提出了以下方法:一种基于正确证明行序列识别验证"证明谜题"(Proof Puzzles)的新算法,与现有基于编辑距离的算法相比,该算法提升了公平性;一个通过遍历抽象语法树(abstract syntax tree)来检查源代码中任务相关特征的开源静态分析工具;一种用于定义正则表达式的高级语言及开源工具,可降低创建复杂正则表达式的错误率。我们的研究成果嵌入在一份将纸质考试转化为电子考试的完整经验报告中。通过分析评分过程中的自动化程度、收集学生意见并批判性审视自身经验,我们对生成的电子考试进行了评估。结果表明:几乎所有任务均可至少实现部分自动化评分(正确解决方案几乎总能被准确识别);学生认同电子考试是该课程适合的考核形式,但在能否像纸质考试一样充分表达想法方面存在分歧;考官在享受更省时评分流程的同时,考试结果的分数分布与纸质考试几乎完全一致。