The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks. For example, recent work in programming education has used LLMs to generate learning resources, improve error messages, and provide feedback on code. However, one factor that limits progress within the field is that much of the research uses bespoke datasets and different evaluation metrics, making direct comparisons between results unreliable. Thus, there is a pressing need for standardization and benchmarks that facilitate the equitable comparison of competing approaches. One task where LLMs show great promise is program repair, which can be used to provide debugging support and next-step hints to students. In this article, we propose a novel educational program repair benchmark. We curate two high-quality publicly available programming datasets, present a unified evaluation procedure introducing a novel evaluation metric rouge@k for approximating the quality of repairs, and evaluate a set of five recent models to establish baseline performance.

翻译：大语言模型（LLMs）的出现因其在教育任务中的潜在应用而引发了广泛兴趣。例如，编程教育领域的最新研究已利用LLMs生成学习资源、改进错误信息并提供代码反馈。然而，该领域进展受限的一个关键因素是：许多研究采用定制化数据集和不同的评估指标，导致结果间的直接比较不可靠。因此，亟需标准化框架与基准测试，以促进不同方法之间的公平比较。LLMs极具前景的应用之一是程序修复，这一功能可为学生提供调试支持与下一步提示。本文提出了一种新颖的教育程序修复基准测试。我们整理了两个高质量公开编程数据集，引入了一种统一评估流程（含新型评估指标rouge@k用于近似衡量修复质量），并评估了五种现有模型以建立基线性能。