We introduce KorMedMCQA, the first Korean multiple-choice question answering (MCQA) benchmark derived from Korean healthcare professional licensing examinations, covering from the year 2012 to year 2023. This dataset consists of a selection of questions from the license examinations for doctors, nurses, and pharmacists, featuring a diverse array of subjects. We conduct baseline experiments on various large language models, including proprietary/open-source, multilingual/Korean-additional pretrained, and clinical context pretrained models, highlighting the potential for further enhancements. We make our data publicly available on HuggingFace (https://huggingface.co/datasets/sean0042/KorMedMCQA) and provide a evaluation script via LM-Harness, inviting further exploration and advancement in Korean healthcare environments.

翻译：我们提出KorMedMCQA，这是首个源自韩国医疗专业人员执业资格考试（覆盖2012年至2023年）的韩语多选题问答（MCQA）基准。该数据集包含选自医生、护士和药师资格考试的问题，涵盖多种学科领域。我们在多种大型语言模型上进行基准实验，包括专有/开源模型、多语言/韩语附加预训练模型以及临床语境预训练模型，凸显了进一步提升的潜力。我们的数据已通过HuggingFace（https://huggingface.co/datasets/sean0042/KorMedMCQA）公开发布，并通过LM-Harness提供评估脚本，以期在韩国医疗环境中推动进一步的探索与进步。