The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. We make MedAlign available under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.

翻译：大语言模型（LLM）以类人流畅度遵循自然语言指令的能力，为医疗健康领域减少行政负担、提升护理质量提供了诸多机遇。然而，在医疗健康场景中评估LLM的真实文本生成任务仍面临挑战。现有基于电子健康记录（EHR）数据的问答数据集，难以捕捉临床医生面临的信息需求复杂性与文书工作负担。为应对这些挑战，我们提出MedAlign——一个包含983条EHR数据自然语言指令的基准数据集。MedAlign由15名临床医生（涵盖7个专科）整理，包含303条指令的临床医生撰写参考答案，并提供276份纵向EHR数据作为指令-响应对的支撑。我们利用MedAlign评估了6个通用领域LLM，由临床医生对各模型响应的准确性和质量进行排序。研究发现高错误率（GPT-4为35%，MPT-7B-Instruct达68%），且GPT-4在上下文长度从32k降至2k时准确性下降8.3%。最后，我们报告临床医生排序与自动化自然语言生成指标之间的相关性，作为无需人工审核即可对LLM排序的方法。我们根据研究数据使用协议公开发布MedAlign，以支持基于临床医生需求与偏好的LLM任务评估。