Inaccuracies in existing or generated clinical text may lead to serious adverse consequences, especially if it is a misdiagnosis or incorrect treatment suggestion. With Large Language Models (LLMs) increasingly being used across diverse healthcare applications, comprehensive evaluation through dedicated benchmarks is crucial. However, such datasets remain scarce, especially across diverse languages and contexts. In this paper, we introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction, developed under the guidance of experienced clinicians. Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese, with natural clinical cases annotated and reviewed by domain experts. We assessed the performance of a range of general-purpose, language-specific, and medical-domain language models across all three tasks. Our results reveal notable performance gaps, particularly in non-English settings, highlighting the need for clinically grounded, language-aware systems. By making MedErrBench and our evaluation protocols publicly-available, we aim to advance multilingual clinical NLP to promote safer and more equitable AI-based healthcare globally. The dataset is available in the supplementary material. An anonymized version of the dataset is available at: https://github.com/congboma/MedErrBench.

翻译：现有或生成的临床文本中的不准确信息可能导致严重的负面后果，尤其是当涉及误诊或错误治疗建议时。随着大语言模型（LLM）在多样化医疗健康应用中的日益普及，通过专用基准进行全面评估至关重要。然而，此类数据集仍然稀缺，尤其是在不同语言和临床情境下。本文介绍了MedErrBench，这是首个在经验丰富的临床医生指导下开发的、用于错误检测、定位与修正的多语言基准。基于扩展的十类常见错误分类体系，MedErrBench涵盖英语、阿拉伯语和中文，所有自然临床案例均由领域专家标注与审核。我们评估了一系列通用模型、语言专用模型及医疗领域语言模型在所有三项任务上的表现。结果显示存在显著的性能差距，尤其在非英语场景中，这凸显了对基于临床实践且具备语言感知能力的系统的迫切需求。通过公开MedErrBench数据集及评估协议，我们旨在推动多语言临床自然语言处理的发展，以促进全球更安全、更公平的基于人工智能的医疗健康服务。数据集已附于补充材料中。匿名版本数据集可通过以下链接获取：https://github.com/congboma/MedErrBench。