The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

翻译：AI助手的整合，尤其是通过大型语言模型（LLMs）的发展，已引发了计算机科学教育领域的广泛讨论。新兴研究开始探索LLMs在教育中的应用，但少有研究考察LLMs对入门级编程课程学生的影响，尤其是在真实场景和长期跨度下的效果。为填补这一研究空白，我们开展了一项为期一学期的受试者间研究，共有50名学生参与，使用了由我们研究团队开发的LLM辅助工具CodeTutor。研究结果表明，使用CodeTutor的学生（实验组）在期末成绩上相比未使用该工具的学生（对照组）取得了统计上显著的提升。在实验组内部，此前没有LLM辅助工具使用经验的学生表现出更大的成绩提升幅度。我们还发现，学生对CodeTutor的能力给予了积极反馈，但也对其在培养批判性思维方面的作用有限表示担忧。随着学期推进，学生对CodeTutor建议的认同度逐渐下降，而对传统人类助教支持的偏好日益增强。进一步分析显示，用户提示词的质量与CodeTutor的响应有效性显著相关。基于我们的结果，我们讨论了将生成式AI素养融入课程以培养批判性思维的启示，并考察了用户与LLM辅助工具互动的时间动态变化。我们还探讨了工具预期功能与学生实际能力之间的差距，这揭示了需要制定针对性策略以改善教育成果。