The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establishing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.

翻译：各类医学大语言模型在医疗领域的涌现凸显了统一评估标准的必要性，因为人工评估大语言模型耗时耗力。针对这一问题，我们提出了 MedBench——一个面向中文医疗领域的综合性基准测试集，包含来自多学科真实考试题目和医疗报告的40,041道问题。具体而言，该基准由四大核心模块构成：中国执业医师资格考试、住院医师规范化培训考试、主治医师职称考试以及涵盖检查、诊断和治疗全流程的临床真实病例。MedBench 再现了中国大陆医生的教育进阶与临床实践历程，从而成为评估医学语言模型知识掌握程度与推理能力的可信基准。我们开展了大量实验并从多维度进行深入分析，得出以下发现：(1) 中文医学大语言模型在此基准上表现欠佳，表明其在临床知识与诊断精确性方面亟需重大突破；(2) 若干通用领域大语言模型意外具备可观的医学知识。这些发现既揭示了 MedBench 体系下大语言模型的能力边界，也明确了其局限性，最终目标是为医学研究社区提供助力。