Large Language Models have shown promising results in their ability to encode general medical knowledge in standard medical question-answering datasets. However, their potential application in clinical practice requires evaluation in domain-specific tasks, where benchmarks are largely missing. In this study semioLLM, we test the ability of state-of-the-art LLMs (GPT-3.5, GPT-4, Mixtral 8x7B, and Qwen-72chat) to leverage their internal knowledge and reasoning for epilepsy diagnosis. Specifically, we obtain likelihood estimates linking unstructured text descriptions of seizures to seizure-generating brain regions, using an annotated clinical database containing 1269 entries. We evaluate the LLM's performance, confidence, reasoning, and citation abilities in comparison to clinical evaluation. Models achieve above-chance classification performance with prompt engineering significantly improving their outcome, with some models achieving close-to-clinical performance and reasoning. However, our analyses also reveal significant pitfalls with several models being overly confident while showing poor performance, as well as exhibiting citation errors and hallucinations. In summary, our work provides the first extensive benchmark comparing current SOTA LLMs in the medical domain of epilepsy and highlights their ability to leverage unstructured texts from patients' medical history to aid diagnostic processes in health care.

翻译：大型语言模型在标准医学问答数据集上展现出了编码通用医学知识的良好能力。然而，其在临床实践中的潜在应用需要在领域特定任务中进行评估，而此类任务的基准测试目前尚属缺乏。在本研究SemioLLM中，我们测试了前沿大型语言模型（GPT-3.5、GPT-4、Mixtral 8x7B和Qwen-72chat）利用其内部知识与推理进行癫痫诊断的能力。具体而言，我们使用包含1269条标注条目的临床数据库，获取了将非结构化的癫痫发作文本描述与致痫脑区相关联的似然估计。我们评估了这些模型在性能、置信度、推理及引用能力方面与临床评估的对比。通过提示工程显著改善结果后，模型实现了高于随机水平的分类性能，部分模型达到了接近临床水平的性能与推理能力。然而，我们的分析也揭示了显著缺陷：多个模型在表现不佳时仍表现出过度自信，同时存在引用错误和幻觉现象。总之，我们的工作首次在癫痫医学领域建立了广泛基准以比较当前最先进的大型语言模型，并突显了其利用患者病史中的非结构化文本来辅助医疗诊断流程的能力。