Open-ended coding tasks, which ask students to construct programs according to certain specifications, are common in computer science education. Student modeling can be challenging since their open-ended nature means that student code can be diverse. Traditional knowledge tracing (KT) models that only analyze response correctness may not fully capture nuances in student knowledge from student code. In this paper, we introduce Test case-Informed Knowledge Tracing for Open-ended Coding (TIKTOC), a framework to simultaneously analyze and predict both open-ended student code and whether the code passes each test case. We augment the existing CodeWorkout dataset with the test cases used for a subset of the open-ended coding questions, and propose a multi-task learning KT method to simultaneously analyze and predict 1) whether a student's code submission passes each test case and 2) the student's open-ended code, using a large language model as the backbone. We quantitatively show that these methods outperform existing KT methods for coding that only use the overall score a code submission receives. We also qualitatively demonstrate how test case information, combined with open-ended code, helps us gain fine-grained insights into student knowledge.

翻译：开放式编程任务要求学生根据特定规范构建程序，在计算机科学教育中十分常见。由于开放式特性导致学生代码具有多样性，学生建模面临挑战。传统知识追踪模型仅分析作答正确性，可能无法从学生代码中充分捕捉知识掌握的细微差异。本文提出面向开放式编程的测试用例知识追踪框架，该框架能同步分析与预测开放式学生代码及其通过各测试用例的情况。我们通过为部分开放式编程题目添加测试用例来扩展现有CodeWorkout数据集，并提出基于大语言模型的多任务学习知识追踪方法，该方法能同步分析与预测：1）学生代码提交是否通过各测试用例；2）学生的开放式代码。定量实验表明，这些方法优于仅使用代码提交总分的现有编程知识追踪方法。我们还通过定性分析展示了测试用例信息与开放式代码相结合如何帮助获取学生知识的细粒度洞察。