Feedback is a critical aspect of improvement. Unfortunately, when there is a lot of feedback from multiple sources, it can be difficult to distill the information into actionable insights. Consider student evaluations of teaching (SETs), which are important sources of feedback for educators. They can give instructors insights into what worked during a semester. A collection of SETs can also be useful to administrators as signals for courses or entire programs. However, on a large scale as in high-enrollment courses or administrative records over several years, the volume of SETs can render them difficult to analyze. In this paper, we discuss a novel method for analyzing SETs using natural language processing (NLP) and large language models (LLMs). We demonstrate the method by applying it to a corpus of 5,000 SETs from a large public university. We show that the method can be used to extract, embed, cluster, and summarize the SETs to identify the themes they express. More generally, this work illustrates how to use the combination of NLP techniques and LLMs to generate a codebook for SETs. We conclude by discussing the implications of this method for analyzing SETs and other types of student writing in teaching and research settings.

翻译：反馈是改进的关键环节，然而当来自多个来源的反馈数量庞大时，将信息提炼为可操作的见解往往颇具挑战。以学生教学评价（SETs）为例，这类评价是教育工作者获取反馈的重要渠道，能够帮助教师了解学期中教学成效的亮点。同时，SETs的集合也可作为课程乃至整个项目成效的信号，为管理者提供参考。但面对大规模场景——例如高注册人数的课程或跨年度行政记录——海量的SETs使得分析工作变得困难。本文探讨了一种结合自然语言处理（NLP）与大语言模型（LLMs）分析SETs的创新方法。我们通过将该方法应用于某大型公立大学5000份SETs语料库进行实证研究，展示其如何通过提取、嵌入、聚类与总结SETs内容，识别其中表达的主题。更广泛而言，本研究阐明了如何利用NLP技术与LLMs的协同作用，为SETs生成编码手册。最后，我们讨论了该方法在教学与研究场景中分析SETs及其他类型学生写作文本的应用前景。