LLMs promise to democratize technical work in complex domains like programmatic data analysis, but not everyone benefits equally. We study how students with varied experiences use LLMs to complete Python-based data analysis in computational notebooks in a graduate course. Drawing on homework logs, recordings, and surveys from 36 students, we ask: Which experience matters most, and how does it shape AI use? Our mixed-methods analysis shows that technical experience -- not AI familiarity or communication skills -- remains a significant predictor of success. Students also vary widely in how they leverage LLMs, struggling at stages of forming intent, expressing inputs, interpreting outputs, and assessing results. We identify success and failure behaviors, such as providing context or decomposing prompts, that distinguish effective use. These findings inform AI literacy interventions, highlighting that lightweight demonstrations improve surface fluency but are insufficient; deeper training and scaffolds are needed to cultivate resilient AI use skills.
翻译:大语言模型有望在程序化数据分析等复杂领域实现技术工作的民主化,但并非所有人都能平等获益。本研究探讨了在研究生课程中,具有不同经验的学生如何利用大语言模型在计算笔记本中完成基于Python的数据分析。通过分析36名学生的作业日志、录音记录与问卷调查,我们提出核心问题:何种经验最为关键?它如何塑造人工智能的使用方式?我们的混合方法分析表明,技术经验——而非对人工智能的熟悉程度或沟通技巧——仍是预测成功的重要指标。学生在利用大语言模型的方式上也存在显著差异,他们在形成意图、表达输入、解读输出和评估结果等阶段均面临挑战。我们识别了区分有效使用的成功与失败行为模式,例如提供上下文或分解提示等。这些发现为人工智能素养培养提供了实践依据,表明轻量级演示虽能提升表面流畅度,但远远不足;需要更深入的训练与支架式教学,才能培养出稳健的人工智能应用能力。