It is important for researchers to understand precisely how data scientists turn raw data into insights, including typical programming patterns, workflow, and methodology. This paper contributes a novel system, called DataInquirer, that tracks incremental code executions in Jupyter notebooks (a type of computational notebook). The system allows us to quantitatively measure timing, workflow, and operation frequency in data science tasks without resorting to human annotation or interview. In a series of pilot studies, we collect 97 traces, logging data scientist activities across four studies. While this paper presents a general system and data analysis approach, we focus on a foundational sub-question in our pilot studies: How consistent are different data scientists in analyzing the same data? We taxonomize variation between data scientists on the same dataset according to three categories: semantic, syntactic, and methodological. Our results suggest that there are statistically significant differences in the conclusions reached by different data scientists on the same task and present quantitative evidence for this phenomenon. Furthermore, our results suggest that AI-powered code tools subtly influence these results, allowing student participants to generate workflows that more resemble expert data practitioners.

翻译：研究人员需要精确理解数据科学家如何将原始数据转化为洞见，包括典型的编程模式、工作流程与方法论。本文提出了一种名为DataInquirer的新型系统，该系统能够追踪Jupyter notebook（一种计算型笔记本）中的增量代码执行过程。该系统使我们能够定量测量数据科学任务中的时间消耗、工作流程与操作频率，而无需依赖人工标注或访谈。通过一系列试点研究，我们收集了97条轨迹，记录了四项研究中数据科学家的活动。尽管本文提出了通用系统与数据分析方法，我们在试点研究中重点关注一个基础性子问题：不同数据科学家在分析相同数据时的一致性如何？我们依据语义、句法和方法论三个类别，对同一数据集上不同数据科学家之间的差异进行了分类。研究结果表明，不同数据科学家在同一任务中得出的结论存在统计学上的显著差异，并为此现象提供了量化证据。此外，我们的研究还表明，基于人工智能的代码工具会微妙地影响这些结果，使学生参与者能够生成更接近专家数据从业者的工作流程。