The recently unprecedented advancements in Large Language Models (LLMs) have propelled the medical community by establishing advanced medical-domain models. However, due to the limited collection of medical datasets, there are only a few comprehensive benchmarks available to gauge progress in this area. In this paper, we introduce a new medical question-answering (QA) dataset that contains massive manual instruction for solving Traditional Chinese Medicine examination tasks, called TCMD. Specifically, our TCMD collects massive questions across diverse domains with their annotated medical subjects and thus supports us in comprehensively assessing the capability of LLMs in the TCM domain. Extensive evaluation of various general LLMs and medical-domain-specific LLMs is conducted. Moreover, we also analyze the robustness of current LLMs in solving TCM QA tasks by introducing randomness. The inconsistency of the experimental results also reveals the shortcomings of current LLMs in solving QA tasks. We also expect that our dataset can further facilitate the development of LLMs in the TCM area.

翻译：近期，大语言模型（LLM）取得了前所未有的进展，通过建立先进的医疗领域模型，推动了医学界的发展。然而，由于医学数据集的收集有限，目前仅有少数综合性基准可用于衡量该领域的进展。本文介绍了一个新的医学问答数据集，该数据集包含了用于解决传统中医考试任务的大量人工指令，称为TCMD。具体而言，我们的TCMD收集了跨多个领域的大量问题及其标注的医学主题，从而支持我们全面评估LLM在中医领域的能力。我们对多种通用LLM和医疗领域专用LLM进行了广泛评估。此外，我们还通过引入随机性，分析了当前LLM在解决中医问答任务时的鲁棒性。实验结果的不一致性也揭示了当前LLM在解决问答任务方面的不足。我们也期望我们的数据集能进一步促进LLM在中医领域的发展。