Personalized chatbot-based teaching assistants can be crucial in addressing increasing classroom sizes, especially where direct teacher presence is limited. Large language models (LLMs) offer a promising avenue, with increasing research exploring their educational utility. However, the challenge lies not only in establishing the efficacy of LLMs but also in discerning the nuances of interaction between learners and these models, which impact learners' engagement and results. We conducted a formative study in an undergraduate computer science classroom (N=145) and a controlled experiment on Prolific (N=356) to explore the impact of four pedagogically informed guidance strategies on the learners' performance, confidence and trust in LLMs. Direct LLM answers marginally improved performance, while refining student solutions fostered trust. Structured guidance reduced random queries as well as instances of students copy-pasting assignment questions to the LLM. Our work highlights the role that teachers can play in shaping LLM-supported learning environments.

翻译：个性化聊天机器人助教在应对日益扩大的班级规模中具有关键作用，尤其是在教师直接指导有限的情况下。大语言模型（LLMs）为此提供了富有前景的途径，相关教育应用研究日益增多。然而，挑战不仅在于验证LLMs的有效性，更在于厘清学习者与这些模型之间交互的细微差异——这种交互直接影响着学习者的参与度与学习成效。我们在计算机科学本科课堂（N=145）和Prolific平台受控实验（N=356）中开展形成性研究，探究四种基于教学理论的引导策略对学习者表现、信心及对LLM信任度的影响。直接提供LLM答案对成绩提升作用有限，而完善学生解决方案则增强了信任感。结构化引导减少了随机查询以及学生将作业题目直接复制粘贴给LLM的行为。本研究凸显了教师在塑造LLM支持型学习环境中的关键作用。