To meet the ever-increasing demands of the cybersecurity workforce, AI tutors have been proposed for personalized, scalable education. But, while AI tutors have shown promise in introductory programming courses, no work has evaluated their use in hands-on exploration and exploitation of systems (e.g., ``capture-the-flag'') commonly used to teach cybersecurity. Thus, despite growing interest and need, no work has evaluated how students use AI tutors or whether they benefit from their presence in real, large-scale cybersecurity courses. To answer this, we conducted a semester-long observational study on the use of an embedded AI tutor with 309 students in an upper-division introductory cybersecurity course. By analyzing 142,526 student queries sent to the AI tutor across 396 cybersecurity challenges spanning 9 core cybersecurity topics and an accompanying set of post-semester surveys, we find (1) what queries and conversational strategies students use with AI tutors, (2) how these strategies correlate with challenge completion, and (3) students' perceptions of AI tutors in cybersecurity education. In particular, we identify three broad AI tutor conversational styles among users: Short (bounded, few-turn exchanges), Reactive (repeatedly submitting code and errors), and Proactive (driving problem-solving through targeted inquiry). We also find that the use of these styles significantly predicts challenge completion, and that this effect increases as materials become more advanced. Furthermore, students valued the tutor's availability but reported that it became less useful for harder material. Based on this, we provide suggestions for security educators and developers on practical AI tutor use.
翻译:为满足网络安全领域日益增长的人才需求,人工智能导师被提出用于实现个性化、可扩展的教育。然而,尽管AI导师在入门级编程课程中已展现出潜力,目前尚无研究评估其在网络安全教学中常用的动手探索与系统利用实践(例如“夺旗赛”)中的应用效果。因此,尽管关注度与需求不断增长,仍未有研究评估学生如何实际使用AI导师,或他们在真实、大规模的网络安全课程中是否从中受益。为解答这一问题,我们在一个面向高阶学生的网络安全入门课程中,对309名学生使用嵌入式AI导师的情况进行了为期一学期的观察性研究。通过分析学生在涵盖9个核心网络安全主题的396项挑战中向AI导师发送的142,526条查询,并结合学期末的问卷调查,我们发现了:(1)学生向AI导师提出何种查询及采用何种对话策略;(2)这些策略如何与挑战完成度相关联;(3)学生对AI导师在网络安全教育中的看法。具体而言,我们识别出用户中三种广泛的AI导师对话风格:简短型(有限轮次、简短的交流)、反应型(反复提交代码和错误信息)以及主动型(通过有针对性的提问推动问题解决)。我们还发现,这些风格的使用能显著预测挑战完成情况,且随着学习材料难度提升,这种效应会增强。此外,学生重视导师的随时可用性,但也反映其在面对更难的材料时帮助有限。基于这些发现,我们为安全领域教育者及开发者提供了关于AI导师实际应用的建议。