The recent explosion in popularity of large language models (LLMs) has inspired learning engineers to incorporate them into adaptive educational tools that automatically score summary writing. Understanding and evaluating LLMs is vital before deploying them in critical learning environments, yet their unprecedented size and expanding number of parameters inhibits transparency and impedes trust when they underperform. Through a collaborative user-centered design process with several learning engineers building and deploying summary scoring LLMs, we characterized fundamental design challenges and goals around interpreting their models, including aggregating large text inputs, tracking score provenance, and scaling LLM interpretability methods. To address their concerns, we developed iScore, an interactive visual analytics tool for learning engineers to upload, score, and compare multiple summaries simultaneously. Tightly integrated views allow users to iteratively revise the language in summaries, track changes in the resulting LLM scores, and visualize model weights at multiple levels of abstraction. To validate our approach, we deployed iScore with three learning engineers over the course of a month. We present a case study where interacting with iScore led a learning engineer to improve their LLM's score accuracy by three percentage points. Finally, we conducted qualitative interviews with the learning engineers that revealed how iScore enabled them to understand, evaluate, and build trust in their LLMs during deployment.

翻译：大型语言模型（LLM）的迅速普及激发了学习工程师将其整合到自适应教育工具中，用于自动评分摘要写作。在将LLM部署到关键学习环境前，理解与评估其性能至关重要，然而模型空前的规模与持续增长的参数数量，不仅阻碍了对其内部机制的透明度，更在其表现欠佳时损害了用户信任。通过与多位构建并部署摘要评分LLM的学习工程师开展以用户为中心的协作式设计流程，我们揭示了模型解读中的核心设计挑战与目标，包括聚合大规模文本输入、追踪评分来源以及扩展LLM可解释性方法。为解决上述问题，我们开发了iScore——一款交互式可视化分析工具，支持学习工程师同时上传、评分并比较多个摘要。紧密集成的视图界面使用户能迭代修改摘要措辞、追踪LLM评分变化，并在多抽象层级上可视化模型权重。为验证该方案的有效性，我们让三位学习工程师持续使用iScore一个月。案例分析表明，通过交互操作，一位学习工程师成功将其LLM的评分准确率提升了三个百分点。最终，对学习工程师的定性访谈揭示了iScore如何帮助他们在部署过程中理解、评估并建立对LLM的信任。