There is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding exercises, the use of LLMs for generating programming MCQs has not been extensively explored. We analyzed the capability of GPT-4 to produce multiple-choice questions (MCQs) aligned with specific learning objectives (LOs) from Python programming classes in higher education. Specifically, we developed an LLM-powered (GPT-4) system for generation of MCQs from high-level course context and module-level LOs. We evaluated 651 LLM-generated and 449 human-crafted MCQs aligned to 246 LOs from 6 Python courses. We found that GPT-4 was capable of producing MCQs with clear language, a single correct choice, and high-quality distractors. We also observed that the generated MCQs appeared to be well-aligned with the LOs. Our findings can be leveraged by educators wishing to take advantage of the state-of-the-art generative models to support MCQ authoring efforts.

翻译：教育工作者持续需要开发并维护有效的、与时俱进的评估工具。尽管计算机教育领域围绕利用大语言模型（LLMs）生成和处理编程练习的研究日益增多，但将LLMs用于生成编程类多项选择题（MCQs）的工作尚未得到广泛探索。我们分析了GPT-4生成与高等教育Python编程课程中特定学习目标（LOs）对齐的多项选择题的能力。具体而言，我们开发了一个基于LLM（GPT-4）的系统，能从高层课程背景及模块级学习目标中自动生成MCQs。我们针对6门Python课程中246个学习目标，评估了651道LLM生成和449道人工设计的MCQs。研究发现，GPT-4能够生成语言清晰、单选题正确答案明确且干扰项质量高的MCQs。同时，生成的MCQs与学习目标表现出良好的一致性。我们的发现可为希望借助当前最先进生成模型来支持MCQ编写工作的教育工作者提供参考。