This study introduces EHRNoteQA, a novel patient-specific question answering benchmark tailored for evaluating Large Language Models (LLMs) in clinical environments. Based on MIMIC-IV Electronic Health Record (EHR), a team of three medical professionals has curated the dataset comprising 962 unique questions, each linked to a specific patient's EHR clinical notes. What makes EHRNoteQA distinct from existing EHR-based benchmarks is as follows: Firstly, it is the first dataset to adopt a multi-choice question answering format, a design choice that effectively evaluates LLMs with reliable scores in the context of automatic evaluation, compared to other formats. Secondly, it requires an analysis of multiple clinical notes to answer a single question, reflecting the complex nature of real-world clinical decision-making where clinicians review extensive records of patient histories. Our comprehensive evaluation on various large language models showed that their scores on EHRNoteQA correlate more closely with their performance in addressing real-world medical questions evaluated by clinicians than their scores from other LLM benchmarks. This underscores the significance of EHRNoteQA in evaluating LLMs for medical applications and highlights its crucial role in facilitating the integration of LLMs into healthcare systems. The dataset will be made available to the public under PhysioNet credential access, promoting further research in this vital field.

翻译：本研究提出EHRNoteQA，一个专为评估临床环境中大型语言模型（LLMs）而设计的患者特异性问答基准。基于MIMIC-IV电子健康记录（EHR）数据，由三位医学专业人员组成的团队构建了包含962个独特问题的数据集，每个问题均关联特定患者的EHR临床笔记。EHRNoteQA相较于现有基于EHR的基准具有以下独特性：首先，它是首个采用多项选择问答格式的数据集，相较于其他格式，该设计能在自动评估情境下通过可靠得分有效评估LLMs。其次，每个问题需分析多份临床笔记才能回答，这反映了临床实践中医生需审阅海量患者病史记录的真实决策复杂性。我们对多种大型语言模型的全面评估表明，相较于其他LLM基准测试得分，模型在EHRNoteQA上的得分与临床医生评估其解决真实医学问题表现之间的相关性更为紧密。这突显了EHRNoteQA在评估医学领域LLMs方面的重要性，并彰显其在促进LLMs融入医疗保健系统中的关键作用。该数据集将通过PhysioNet凭据访问向公众开放，以推动这一关键领域的深入研究。