As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all six years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

翻译：随着大语言模型（LLMs）在多语言用户群体中日益普及，我们认为有必要对其进行基准测试，以更深入地理解模型在英语以外语言中的行为、失效模式与局限性。本研究针对近五年（含本年度）的日本国家医师资格考试，评估了LLM应用程序接口（ChatGPT、GPT-3和GPT-4）的表现。研究团队由以日语为母语的自然语言处理研究者与一名在日本执业的认证心脏病专家组成。实验结果表明，GPT-4在性能上优于ChatGPT与GPT-3，且通过了全部六个年度的考试，凸显了LLMs在类型学上与英语差异显著的语言中的潜力。然而，评估也揭示了当前LLM应用程序接口的关键局限性。首先，LLMs有时会选取应在日本医疗实践中严格规避的禁止选项，例如建议安乐死。此外，分析显示，由于管线中非拉丁文字符的标记化处理方式，日语版本的API成本普遍更高且最大上下文窗口更小。我们以"Igaku QA"为名称公开发布该基准测试集，同时附带所有模型输出与考试元数据。希望我们的研究成果与基准测试能够推动LLM在更多元化应用场景中的发展。基准测试集可通过https://github.com/jungokasai/IgakuQA获取。