In recent years, there have been significant breakthroughs in the field of natural language processing, particularly with the development of large language models (LLMs). These LLMs have showcased remarkable capabilities on various benchmarks. In the healthcare field, the exact role LLMs and other future AI models will play remains unclear. There is a potential for these models in the future to be used as part of adaptive physician training, medical co-pilot applications, and digital patient interaction scenarios. The ability of AI models to participate in medical training and patient care will depend in part on their mastery of the knowledge content of specific medical fields. This study investigated the medical knowledge capability of LLMs, specifically in the context of internal medicine subspecialty multiple-choice test-taking ability. We compared the performance of several open-source LLMs (Koala 7B, Falcon 7B, Stable-Vicuna 13B, and Orca Mini 13B), to GPT-4 and Claude 2 on multiple-choice questions in the field of Nephrology. Nephrology was chosen as an example of a particularly conceptually complex subspecialty field within internal medicine. The study was conducted to evaluate the ability of LLM models to provide correct answers to nephSAP (Nephrology Self-Assessment Program) multiple-choice questions. The overall success of open-sourced LLMs in answering the 858 nephSAP multiple-choice questions correctly was 17.1% - 25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas GPT-4 achieved a score of 73.3%. We show that current widely used open-sourced LLMs do poorly in their ability for zero-shot reasoning when compared to GPT-4 and Claude 2. The findings of this study potentially have significant implications for the future of subspecialty medical training and patient care.

翻译：近年来，自然语言处理领域取得了重大突破，尤其是大型语言模型的发展。这些大型语言模型在各类基准测试中展现出卓越能力。在医疗健康领域，大型语言模型及其他未来AI模型将扮演何种角色尚不明确。这些模型未来可能应用于适应性医生培训、医疗辅助应用及数字化患者交互场景。AI模型参与医学培训与患者护理的能力，部分取决于其对特定医学领域知识内容的掌握程度。本研究探究了大型语言模型在医学知识方面的能力，特别是内科学专科多选题作答能力。我们比较了多个开源大型语言模型（Koala 7B、Falcon 7B、Stable-Vicuna 13B和Orca Mini 13B）与GPT-4及Claude 2在肾脏病学领域多选题上的表现。选择肾脏病学作为内科学中概念高度复杂的专科领域代表。研究旨在评估大型语言模型正确解答nephSAP（肾脏病学自评计划）多选题的能力。开源自大型语言模型在858道nephSAP多选题中的总正确率为17.1%至25.5%。相比之下，Claude 2的正确率为54.4%，GPT-4则达到了73.3%。研究结果表明，当前广泛使用的开源大型语言模型在零样本推理能力上明显弱于GPT-4与Claude 2。该研究结果可能对未来的专科医疗培训与患者护理产生重要影响。