In feedback generation for logical errors in programming assignments, large language model (LLM)-based methods have shown great promise. These methods ask the LLM to generate feedback given the problem statement and a student's (buggy) submission. There are several issues with these types of methods. First, the generated feedback messages are often too direct in revealing the error in the submission and thus diminish valuable opportunities for the student to learn. Second, they do not consider the student's learning context, i.e., their previous submissions, current knowledge, etc. Third, they are not layered since existing methods use a single, shared prompt for all student submissions. In this paper, we explore using LLMs to generate a "feedback-ladder", i.e., multiple levels of feedback for the same problem-submission pair. We evaluate the quality of the generated feedback-ladder via a user study with students, educators, and researchers. We have observed diminishing effectiveness for higher-level feedback and higher-scoring submissions overall in the study. In practice, our method enables teachers to select an appropriate level of feedback to show to a student based on their personal learning context, or in a progressive manner to go more detailed if a higher-level feedback fails to correct the student's error.

翻译：在编程作业中针对逻辑错误的反馈生成方面，基于大语言模型（LLM）的方法已展现出巨大潜力。这些方法通过向LLM提供问题描述及学生（存在缺陷的）提交代码来生成反馈。然而，这类方法存在若干问题：首先，生成的反饋信息往往过于直接地揭示提交代码中的错误，从而削弱了学生宝贵的学习机会；其次，这些方法未考虑学生的学习背景（如历史提交记录、现有知识水平等）；最后，由于现有方法对所有学生提交均使用单一共享提示词，导致反馈缺乏层次性。本文探索利用LLM生成“反馈阶梯”，即针对同一问题-提交代码对生成多级反馈。我们通过包含学生、教育工作者和研究人员的用户研究评估了所生成反馈阶梯的质量。研究发现，随着反馈层级提高及提交代码得分上升，反馈有效性呈现递减趋势。在实践中，该方法允许教师根据学生个人学习情况选择合适的反馈层级，或采用渐进式策略——当高层级反馈未能纠正学生错误时，逐步提供更详细的反馈。