We evaluated the capability of generative pre-trained transformers (GPT), to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. Discussions of potential uses (e.g., exercise generation, code explanation) and misuses (e.g., cheating) of this emerging technology in programming education have intensified, but to date there has not been a rigorous analysis of the models' capabilities in the realistic context of a full-fledged programming course with diverse set of assessment instruments. We evaluated GPT on three Python courses that employ assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Further, we studied if and how successfully GPT models leverage feedback provided by an auto-grader. We found that the current models are not capable of passing the full spectrum of assessments typically involved in a Python programming course (<70% on even entry-level modules). Yet, it is clear that a straightforward application of these easily accessible models could enable a learner to obtain a non-trivial portion of the overall available score (>55%) in introductory and intermediate courses alike. While the models exhibit remarkable capabilities, including correcting solutions based on auto-grader's feedback, some limitations exist (e.g., poor handling of exercises requiring complex chains of reasoning steps). These findings can be leveraged by instructors wishing to adapt their assessments so that GPT becomes a valuable assistant for a learner as opposed to an end-to-end solution.

翻译：我们评估了生成式预训练变换器（GPT）在高等教育阶段初级与中级Python编程课程评估中的通过能力。关于该新兴技术在编程教育中的潜在用途（如习题生成、代码解释）与滥用风险（如作弊）的讨论日益激烈，但迄今尚未有研究在包含多样化评估工具的完整编程课程真实场景中，对模型能力进行严谨分析。针对三门采用从简单选择题（无代码）到需分布式多文件代码库的复杂编程项目（共599道习题）的Python课程，我们评估了GPT的表现。此外，我们研究了GPT模型能否有效利用自动评分系统提供的反馈，及其具体实现方式。研究发现，当前模型无法通过Python编程课程中典型全套评估（即使入门模块得分率也低于70%）。然而显而易见，简单应用这些易获取的模型即可使学习者在入门与中级课程中均获得非显著性总分（>55%）。尽管模型展现出显著能力（包括根据自动评分反馈修正解决方案），仍存在局限性（例如难以处理需复杂推理链的习题）。这些发现可为希望调整课程评估体系的教育工作者提供参考，使GPT成为学习者的有效辅助工具而非端到端的解决方案。