This study introduces EHRNoteQA, a novel patient-specific question answering benchmark tailored for evaluating Large Language Models (LLMs) in clinical environments. Based on MIMIC-IV Electronic Health Record (EHR), a team of three medical professionals has curated the dataset comprising 962 unique questions, each linked to a specific patient's EHR clinical notes. What makes EHRNoteQA distinct from existing EHR-based benchmarks is as follows: Firstly, it is the first dataset to adopt a multi-choice question answering format, a design choice that effectively evaluates LLMs with reliable scores in the context of automatic evaluation, compared to other formats. Secondly, it requires an analysis of multiple clinical notes to answer a single question, reflecting the complex nature of real-world clinical decision-making where clinicians review extensive records of patient histories. Our comprehensive evaluation on various large language models showed that their scores on EHRNoteQA correlate more closely with their performance in addressing real-world medical questions evaluated by clinicians than their scores from other LLM benchmarks. This underscores the significance of EHRNoteQA in evaluating LLMs for medical applications and highlights its crucial role in facilitating the integration of LLMs into healthcare systems. The dataset will be made available to the public under PhysioNet credential access, promoting further research in this vital field.

翻译：本研究提出了EHRNoteQA，一个专为在临床环境中评估大语言模型（LLMs）而设计的患者特定问答基准。基于MIMIC-IV电子健康记录（EHR），由三位医学专业人员组成的小组整理了一个包含962个独特问题的数据集，每个问题均关联至特定患者的EHR临床笔记。EHRNoteQA区别于现有基于EHR的基准之处如下：首先，它是首个采用多项选择问答格式的数据集，在自动评估背景下，与其他格式相比，这种设计选择能有效以可靠分数评估大语言模型。其次，该数据集要求分析多份临床笔记以回答单个问题，这反映了真实临床决策中医生审阅患者广泛病史记录的复杂本质。我们对各类大语言模型的全面评估表明，相较于其他大语言模型基准，这些模型在EHRNoteQA上的得分与临床医生评估的解决真实医学问题表现更为紧密相关。这凸显了EHRNoteQA在评估医学应用大语言模型方面的重要意义，并揭示了其在促进大语言模型融入医疗系统过程中的关键作用。该数据集将在PhysioNet凭据访问权限下向公众开放，以推动这一重要领域的进一步研究。