As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all five years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

翻译：随着大型语言模型（LLMs）在多种语言使用者中日益普及，我们认为有必要对其进行基准测试，以更深入地理解模型在非英语语言中的行为、失败模式及局限性。本研究对过去五年日本国家医学执照考试中的LLM应用程序接口（ChatGPT、GPT-3和GPT-4）进行了评估。研究团队由母语为日语的NLP研究人员及一名在日本执业的执业心脏病专家组成。实验表明，GPT-4在全部五年考试中均优于ChatGPT和GPT-3并通过考核，凸显了LLMs在类型学上与英语相距甚远的语言中的潜力。然而，评估也揭示了当前LLM应用程序接口的关键局限性。首先，LLMs有时会选取在日本医疗实践中应严格禁止的选项（如建议安乐死）。此外，分析显示由于管道对非拉丁字母的当前分词方式，日语的应用程序接口使用成本普遍较高，最大上下文长度也较小。我们将基准测试数据集命名为Igaku QA，并公开所有模型输出及考试元数据。希望我们的结果和基准测试能推动LLMs在更多样化场景中的应用。基准测试数据集详见 https://github.com/jungokasai/IgakuQA。