Large language models (LLMs) are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets -- leading to major blind spots for evaluating clinical systems. In this work, we perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions. Specifically, we train a pre-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment-specific context available before generation. We conduct a prospective analysis of our model over 4.5 months of user feedback, finding that our prediction model achieves an AUROC of 0.719. Further, we estimate the benefit of such predictions in two downstream use cases (guardrail triggering and abstention). Our key conceptual insight is that making use of deployment-specific context (i.e., the provider type, department name, language model used for response), as opposed to only query content, improves the ability to predict whether the user will reject the system output. Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment-specific context, opening the door to targeted guardrails.

翻译：大语言模型（LLMs）正日益集成到临床系统中，使得评估这些系统的现实世界效用变得至关重要。然而，静态基准测试往往衡量的是正确性而非用户接受度，跨查询聚合性能，且需要密集标注的数据集——这导致评估临床系统时存在重大盲点。在本研究中，我们对一个嵌入某学术医疗中心电子健康记录的大语言模型系统进行了以部署为中心的评估，其中用户反馈稀疏但紧密反映了部署条件。具体而言，我们训练了一个响应前分类器，该分类器基于生成前可用的查询内容和部署特定上下文，估计未来交互导致用户拒绝大语言模型响应的风险。我们基于4.5个月的用户反馈对模型进行了前瞻性分析，发现我们的预测模型达到了0.719的AUROC。进一步地，我们在两个下游应用场景（护栏触发和弃权）中估计了此类预测的收益。我们的核心概念见解在于：利用部署特定上下文（即提供者类型、科室名称、用于响应的语言模型），而不仅仅是查询内容，能提高预测用户是否拒绝系统输出的能力。总之，我们的实证案例研究证明了使用部署特定上下文预测用户拒绝的可行性，为针对性护栏的部署打开了大门。