LLMs have become increasingly capable at accomplishing a range of specialized-tasks and can be utilized to expand equitable access to medical knowledge. Most medical LLMs have involved extensive fine-tuning, leveraging specialized medical data and significant, thus costly, amounts of computational power. Many of the top performing LLMs are proprietary and their access is limited to very few research groups. However, open-source (OS) models represent a key area of growth for medical LLMs due to significant improvements in performance and an inherent ability to provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated a range of OS foundation LLMs (7B-70B) on four medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset). We employed a series of prompting strategies, including zero-shot, few-shot, chain-of-thought (random selection and kNN selection), and ensemble/self-consistency voting. We found that OpenMedLM delivers OS SOTA results on three common medical LLM benchmarks, surpassing the previous best performing OS models that leveraged computationally costly extensive fine-tuning. The model delivers a 72.6% accuracy on the MedQA benchmark, outperforming the previous SOTA by 2.4%, and achieves 81.7% accuracy on the MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs which have not yet been documented to date elsewhere, and showcase the benefits of further leveraging prompt engineering to improve the performance of accessible LLMs for medical applications.

翻译：大语言模型在执行一系列专业化任务方面能力日益增强，可用于扩大医疗知识的公平获取。多数医学大语言模型涉及大量微调工作，依赖专业医疗数据和巨大的计算资源（成本高昂）。许多顶尖性能的大语言模型属于专有模型，仅限少数研究团队访问。然而，开源模型因其性能显著提升以及提供医疗领域所需的透明性和合规性的固有能力，成为医学大语言模型发展的关键领域。我们提出OpenMedLM——一种在医学基准测试中为开源大语言模型带来最先进性能的提示平台。我们评估了多个开源基础大语言模型（7B-70B参数）在四项医学基准测试（MedQA、MedMCQA、PubMedQA、MMLU医学子集）上的表现，并采用了一系列提示策略，包括零样本、少样本、思维链（随机选择与k近邻选择）以及集成/自一致性投票。实验表明，OpenMedLM在三个常见医学大语言模型基准测试中达到开源模型最先进结果，超越了此前依赖高计算成本进行深度微调的最佳开源模型。该模型在MedQA基准测试中达到72.6%的准确率，较先前最优模型提升2.4%；在MMLU医学子集中达到81.7%的准确率，成为首个在该基准测试中突破80%准确率的开源大语言模型。我们的结果揭示了开源大语言模型中尚未被文献记载的医学特定涌现特性，并展示了进一步利用提示工程提升可访问大语言模型在医疗应用中的性能优势。