This study examines the impact of Socratic Mind, a Generative Artificial Intelligence (GenAI) powered formative assessment tool that employs Socratic questioning to support student learning in a large, fully online undergraduate-level computing course. Employing a quasi-experimental, mixed-methods design, we investigated participants' engagement patterns, the influence of user experience on engagement, and impacts on both perceived and actual learning outcomes. Data were collected from the system logs, surveys on user experience and perceived engagement and learning gains, student reflections, and course performance data. Results indicated that participants consistently reported high levels of affective, behavioral, and cognitive engagement, and these were strongly linked to positive user experiences and perceived learning outcomes. Quantitative analysis further revealed that students who engaged with the GenAI tool experienced significant gains in their quiz scores compared to those who did not, particularly benefiting students with lower baseline achievement. Additionally, thematic analysis of qualitative feedback revealed substantial perceived improvements in higher-order thinking skills, including problem solving, critical thinking, and self-reflection. Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills. As higher education institutions expand GenAI integration in curriculum, this dialogic, GenAI powered assessment tool can offer a scalable strategy to promote students' meaningful learning outcomes.

翻译：本研究探讨了苏格拉底思维（Socratic Mind）——一种采用苏格拉底式提问法、基于生成式人工智能（GenAI）的形成性评估工具——在一门大规模全在线本科计算课程中对学生学习的支持效果。通过准实验混合方法设计，我们考察了参与者的参与模式、用户体验对参与度的影响，以及对感知与实际学习成果的作用。数据来源于系统日志、用户体验与感知参与度及学习收获的问卷调查、学生反思报告以及课程成绩记录。结果表明，参与者普遍报告了较高水平的情感、行为与认知参与度，且这些维度与积极的用户体验及感知学习成果密切相关。定量分析进一步显示，与未使用该工具的学生相比，使用生成式人工智能工具的学生在测验成绩上取得显著提升，其中基础成绩较低的学生获益尤为明显。此外，定性反馈的主题分析表明，学生在问题解决、批判性思维和自我反思等高阶思维能力方面感受到实质性提升。我们的研究结果凸显了人工智能中介对话在促进深度参与与高阶认知技能方面的潜力。随着高等教育机构在课程中扩大生成式人工智能的整合，这种对话式、基于生成式人工智能的评估工具可为提升学生有意义的学习成果提供可扩展的策略。