We explore the evolving efficacy of three generative pre-trained transformer (GPT) models in generating answers for multiple-choice questions (MCQ) from introductory and intermediate Python programming courses in higher education. We focus on the differences in capabilities of the models prior to the release of ChatGPT (Nov '22), at the time of the release, and today (i.e., Aug '23). Recent studies have established that the abilities of the OpenAI's GPT models to handle assessments originally designed for humans keep increasing as the newer more capable models are released. However, the qualitative differences in the capabilities and limitations of these models to reason about and/or analyze programming MCQs have been under-explored. We evaluated three OpenAI's GPT models on formative and summative MCQ assessments from three Python courses (530 questions) focusing on the qualitative differences in the evolving efficacy of the subsequent models. This study provides further evidence and insight into the trajectory of the current developments where there already exists a technology that can be utilized by students to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments. This study could be leveraged by educators and institutions to better understand the recent technological developments in order to adapt the design of programming assessments as well as to fuel the necessary discussions into how assessments in future programming classes should be updated.

翻译：我们探讨了三种生成式预训练变换器模型在高等教育初、中级Python编程课程中生成多选题答案的能力演进过程。重点关注ChatGPT发布前（2022年11月）、发布时以及当前（即2023年8月）各模型能力的差异。近期研究已证实，随着功能更强的新模型不断发布，OpenAI的GPT模型处理原为人类设计的评估任务的能力持续提升。然而，这些模型在推理和/或分析编程类选择题时，其能力与局限性的质性差异尚未得到充分探索。我们使用三个Python课程（共530道题目）的形成性与总结性选择题评估，对三种OpenAI GPT模型进行了评测，重点分析后续模型能力在演进过程中的质性差异。本研究为当前技术发展轨迹提供了进一步证据与洞见：现有技术已使学生能够不费吹灰之力获得当前被视为有效编程知识与技能评估的及格分数。教育工作者与院校可借助本研究更深入地理解最新技术发展，从而调整编程评估的设计方案，并推动关于未来编程课程评估方式更新必要性的讨论。