This paper introduces a novel approach to create a high-resolution "map" for physics learning: an "atomic" learning objectives (LOs) system designed to capture detailed cognitive processes and concepts required for problem solving in a college-level introductory physics course. Our method leverages Large Language Models (LLMs) for automated labeling of physics questions and introduces a comprehensive set of metrics to evaluate the quality of the labeling outcomes. The atomic LO system, covering nine chapters of an introductory physics course, uses a "subject-verb-object'' structure to represent specific cognitive processes. We apply this system to 131 questions from expert-curated question banks and the OpenStax University Physics textbook. Each question is labeled with 1-8 atomic LOs across three chapters. Through extensive experiments using various prompting strategies and LLMs, we compare automated LOs labeling results against human expert labeling. Our analysis reveals both the strengths and limitations of LLMs, providing insight into LLMs reasoning processes for labeling LOs and identifying areas for improvement in LOs system design. Our work contributes to the field of learning analytics by proposing a more granular approach to mapping learning objectives with questions. Our findings have significant implications for the development of intelligent tutoring systems and personalized learning pathways in STEM education, paving the way for more effective "learning GPS'' systems.
翻译:本文提出了一种为物理学习创建高分辨率"地图"的新方法:一种"原子化"学习目标系统,旨在捕捉大学物理导论课程中解决问题所需的详细认知过程与概念。我们的方法利用大语言模型对物理问题进行自动化标注,并引入一套综合指标来评估标注结果的质量。该原子化学习目标系统覆盖物理导论课程的九个章节,采用"主语-谓语-宾语"结构来表征具体的认知过程。我们将该系统应用于来自专家精选题库和OpenStax大学物理教材的131道题目,每道题目在三个章节范围内被标注1-8个原子化学习目标。通过采用多种提示策略和大语言模型开展大量实验,我们将自动化学习目标标注结果与人类专家标注进行对比。分析揭示了大语言模型的优势与局限,为理解大语言模型标注学习目标的推理过程提供了见解,并指出了学习目标系统设计中需要改进的领域。本研究通过提出更细粒度的学习目标与问题映射方法,为学习分析领域作出贡献。我们的发现对STEM教育中智能辅导系统和个性化学习路径的开发具有重要意义,为构建更高效的"学习导航"系统奠定了基础。