Investigating children's embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination behaviors. Learning scientists have developed Interaction Analysis (IA) methodologies for analyzing such data, but this requires researchers to watch hours of videos to extract and interpret students' learning patterns. Our study aims to simplify researchers' tasks, using Machine Learning and Multimodal Learning Analytics to support the IA processes. Our study combines machine learning algorithms and multimodal analyses to support and streamline researcher efforts in developing a comprehensive understanding of students' scientific engagement through their movements, gaze, and affective responses in a simulated scenario. To facilitate an effective researcher-AI partnership, we present an initial case study to determine the feasibility of visually representing students' states, actions, gaze, affect, and movement on a timeline. Our case study focuses on a specific science scenario where students learn about photosynthesis. The timeline allows us to investigate the alignment of critical learning moments identified by multimodal and interaction analysis, and uncover insights into students' temporal learning progressions.

翻译：研究儿童在混合现实环境中的具身学习（他们在此环境中协作模拟科学过程）需要分析复杂的多模态数据，以解读其学习与协调行为。学习科学家已开发出交互分析方法（IA）用于分析此类数据，但研究者需要观看数小时的视频来提取和解读学生的学习模式。本研究旨在简化研究者的任务，利用机器学习与多模态学习分析支持IA流程。我们结合机器学习算法与多模态分析，辅助并优化研究者全面理解学生在模拟场景中通过动作、注视和情感反应展现的科学参与过程。为促进有效的研究者-人工智能协作，我们通过初步案例研究，验证在时间线上可视化呈现学生状态、动作、注视、情感及运动的可行性。本案例聚焦于学生学习光合作用的特定科学场景。该时间线使我们能够探究多模态分析与交互分析所识别关键学习时刻的一致性，并揭示学生时间性学习进程的深层洞见。