This paper documents a year-long experiment to "profile" the process of learning a programming language: gathering data to understand what makes a language hard to learn, and using that data to improve the learning process. We added interactive quizzes to The Rust Programming Language, the official textbook for learning Rust. Over 13 months, 62,526 readers answered questions 1,140,202 times. First, we analyze the trajectories of readers. We find that many readers drop-out of the book early when faced with difficult language concepts like Rust's ownership types. Second, we use classical test theory and item response theory to analyze the characteristics of quiz questions. We find that better questions are more conceptual in nature, such as asking why a program does not compile vs. whether a program compiles. Third, we performed 12 interventions into the book to help readers with difficult questions. We find that on average, interventions improved quiz scores on the targeted questions by +20%. Fourth, we show that our technique can likely generalize to languages with smaller user bases by simulating our statistical inferences on small N. These results demonstrate that quizzes are a simple and useful technique for understanding language learning at all scales.

翻译：本文记录了一项为期一年的实验，旨在"剖析"编程语言学习的过程：通过收集数据来理解语言学习的难点，并利用这些数据改进学习过程。我们在《Rust编程语言》（The Rust Programming Language，学习Rust的官方教科书）中加入了交互式测验。在13个月的时间里，62,526名读者回答了1,140,202次问题。首先，我们分析了读者的学习轨迹。我们发现，许多读者在面对诸如Rust的所有权类型等困难语言概念时，会提前放弃学习这本书。其次，我们运用经典测验理论和项目反应理论分析了测验问题的特征。我们发现，更优质的问题在本质上更具概念性，例如询问程序为何编译失败，而非程序能否编译。第三，我们在书中进行了12项干预措施，以帮助读者应对困难问题。我们发现，平均而言，这些干预措施使针对目标问题的测验成绩提高了+20%。第四，我们通过在小样本（N）上模拟统计推断，证明了我们的技术很可能可以推广到用户基数较小的语言。这些结果表明，测验是一种简单而有效的技术，可用于理解各种规模的语言学习过程。