Generative AI chatbots enable personalized problem-solving, but effective learning requires students to self-regulate both how they seek help and how they use AI-generated responses. Considering engagement modes across these two actions reveals nuanced reliance patterns: for example, a student may actively engage in help-seeking by clearly specifying areas of need, yet engage passively in response-use by copying AI outputs, or vice versa. However, existing research lacks systematic tools for jointly capturing engagement across help-seeking and response-use, limiting the analysis of such reliance behaviors. We introduce RelianceScope, an analytical framework that characterizes students' reliance on chatbots during problem-solving. RelianceScope (1) operationalizes reliance into nine patterns based on combinations of engagement modes in help-seeking and response-use, and (2) situates these patterns within a knowledge-context lens that accounts for students' prior knowledge and the instructional significance of knowledge components. Rather than prescribing optimal AI use, the framework enables fine-grained analysis of reliance in open-ended student-AI interactions. As an illustrative application, we applied RelianceScope to analyze chat and code-edit logs from 79 college students in a web programming course. Results show that active help-seeking is associated with active response-use, whereas reliance patterns remain similar across knowledge mastery levels. Students often struggled to articulate their knowledge gaps and to adapt AI responses. Using our annotated dataset as a benchmark, we further demonstrate that large language models can reliably detect reliance during help-seeking and response-use. We conclude by discussing the implications of RelianceScope and the design guidelines for AI-supported educational systems.
翻译:生成式AI聊天机器人能够实现个性化问题解决,但有效学习要求学生既要自我调节求助方式,又要规范使用AI生成回答。通过考察这两个行为维度的参与模式,可以揭示细微的依赖模式:例如,学生可能在求助环节主动参与(清晰说明需求领域),却在答案使用环节被动参与(直接复制AI输出),反之亦然。然而,现有研究缺乏能够系统捕捉求助与答案使用双重参与模式的工具,限制了对这类依赖行为的分析。本文提出RelianceScope分析框架,用于刻画学生在问题解决过程中对聊天机器人的依赖特征。该框架(1)依据求助与答案使用参与模式的组合,将依赖性操作化为九种模式;(2)通过知识情境视角定位这些模式,综合考虑学生先验知识与知识组件的教学重要性。本框架并非规定最优AI使用方式,而是支持对开放式学生-AI互动中的依赖行为进行细粒度分析。作为示例应用,我们将RelianceScope应用于分析79名大学生在网页编程课程中的聊天记录与代码编辑日志。结果表明:主动求助与主动使用答案呈正相关,而依赖模式在不同知识掌握水平间保持相似。学生常难以清晰表述知识缺口及调整AI生成回答。基于我们标注的数据集作为基准,进一步证明大语言模型能够可靠检测求助与答案使用过程中的依赖行为。最后,我们讨论了RelianceScope的启示意义及AI辅助教育系统的设计准则。