Ever since the emergence of large language models (LLMs) and related applications, such as ChatGPT, its performance and error analysis for programming tasks have been subject to research. In this work-in-progress paper, we explore the potential of such LLMs for computing educators and learners, as we analyze the feedback it generates to a given input containing program code. In particular, we aim at (1) exploring how an LLM like ChatGPT responds to students seeking help with their introductory programming tasks, and (2) identifying feedback types in its responses. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT along with questions required to elicit feedback and correct solutions. The results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors, which means that students can potentially benefit. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.

翻译：自大型语言模型（LLMs）及相关应用（如ChatGPT）出现以来，其在编程任务中的表现和错误分析一直是研究重点。在这篇进展性论文中，我们探讨了此类LLMs对计算机教育者和学习者的潜在价值，分析了其对包含程序代码的给定输入所生成的反馈。具体而言，我们旨在：（1）探索像ChatGPT这样的LLMs如何回应在入门编程任务中寻求帮助的学生；（2）识别其响应中的反馈类型。为实现这些目标，我们使用从CS1课程收集的数据集中的学生编程序列作为ChatGPT的输入，同时附加用于引发反馈和正确解决方案的问题。结果表明，ChatGPT在处理部分入门编程任务和学生错误时表现尚可，这意味着学生可能从中获益。然而，教育者应提供关于如何使用所生成反馈的指导，因为这些反馈可能包含误导初学者的信息。