The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. We make MedAlign available under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.

翻译：大型语言模型（LLMs）以人类水平的流畅性遵循自然语言指令的能力，为医疗领域减轻行政负担、提升护理质量带来了诸多机遇。然而，在医疗场景下评估LLM执行真实文本生成任务仍面临挑战。现有基于电子健康记录（EHR）的问答数据集未能捕捉临床医生在信息需求和文书负担方面的复杂性。为解决这些问题，我们提出MedAlign——包含983条针对EHR数据的自然语言指令的基准数据集。该数据集由15名临床医生（涵盖7个专科）整理，包含303条指令的临床医生撰写参考答案，并提供276份纵向EHR记录作为指令-响应对的基准。我们利用MedAlign评估了6个通用领域LLM，由临床医生对各模型响应的准确性和质量进行排序。研究发现错误率居高不下，范围从35%（GPT-4）到68%（MPT-7B-Instruct），且GPT-4在从32k上下文长度降至2k时准确率下降8.3%。最后，我们报告了临床医生排名与自动化自然语言生成指标之间的相关性，作为无需人工审核即可对LLM进行排序的方法。我们通过研究数据使用协议公开MedAlign，以支持对符合临床医生需求与偏好的任务进行LLM评估。