The widespread availability of large language models (LLMs) has changed how students engage with coding and problem-solving. While these tools may increase student productivity, they also make it more difficult for instructors to assess students' learning and effort. In this quasi-longitudinal study, we analyze five years of student source code submissions in a graduate-level cloud computing course, focusing on an assignment that remained unchanged and examining students' behavior during the period spanning five semesters before the release of ChatGPT and five semesters after. Student coding behavior has changed significantly since Fall 2022. The length of their final submissions increased. Between consecutive submissions, average edit distances increased while average score improvement decreased, suggesting that both student productivity and learning have decreased after ChatGPT's release. Additionally, there are statistically significant correlations between these behavioral changes and their overall performance. Although we cannot definitively attribute them to LLM misuse, they are consistent with our hypothesis that some students are over-reliant on LLMs, which is negatively affecting their learning outcomes. Our findings raise an alarm around the first generation of graduates in the age of LLMs, calling upon both educators and employers to reflect on their evaluation methods for genuine expertise and productivity.

翻译：大型语言模型（LLMs）的广泛普及改变了学生参与编程和问题解决的方式。虽然这些工具可能提升学生的生产力，但也使得教师评估学生学习成效与努力程度变得更加困难。在这项准纵向研究中，我们分析了一门研究生级别云计算课程中五年间的学生源代码提交记录，重点关注一项保持不变的作业任务，并考察了ChatGPT发布前后各五个学期期间学生的行为变化。自2022年秋季以来，学生的编程行为发生了显著改变：其最终提交的代码长度有所增加；连续提交之间的平均编辑距离增大，而平均分数提升幅度减小，这表明ChatGPT发布后学生的生产力与学习效果均出现下降。此外，这些行为变化与学生的整体表现之间存在统计学上显著的相关性。虽然我们无法明确将其归因于LLM的滥用，但这些现象与我们的假设一致——部分学生过度依赖LLM，这对其学习成果产生了负面影响。我们的研究结果为LLM时代的第一代毕业生敲响了警钟，呼吁教育工作者和雇主共同反思如何评估真正的专业能力与生产力。