Large language models (LLMs) show significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. However, concerns persist regarding the reliability of these benchmarks, which often lack clinical fidelity, robust data management, and safety-oriented evaluation metrics. To address these shortcomings, we introduce MedCheck, the first lifecycle-oriented assessment framework specifically designed for medical benchmarks. Our framework deconstructs a benchmark's development into five continuous stages, from design to governance, and provides a comprehensive checklist of 46 medically-tailored criteria. Using MedCheck, we conducted an in-depth empirical evaluation of 53 medical LLM benchmarks. Our analysis uncovers widespread, systemic issues, including a profound disconnect from clinical practice, a crisis of data integrity due to unmitigated contamination risks, and a systematic neglect of safety-critical evaluation dimensions like model robustness and uncertainty awareness. Based on these findings, MedCheck serves as both a diagnostic tool for existing benchmarks and an actionable guideline to foster a more standardized, reliable, and transparent approach to evaluating AI in healthcare.

翻译：大语言模型在医疗领域展现出巨大潜力，因此催生了大量用于评估其能力的基准测试。然而，人们对这些基准测试的可靠性仍存在担忧，它们通常缺乏临床保真度、稳健的数据管理以及以安全为导向的评估指标。为解决这些缺陷，我们提出了MedCheck——首个专门为医学基准设计的生命周期导向型评估框架。我们的框架将基准测试的开发过程解构为从设计到治理的五个连续阶段，并提供了包含46条医学定制标准的全面核对清单。利用MedCheck，我们针对53个医学大语言模型基准进行了深入的实证评估。我们的分析揭示了广泛存在的系统性问题，包括与临床实践的严重脱节、因未能缓解的数据污染风险而引发的数据完整性危机，以及对鲁棒性和不确定性意识等安全关键评估维度的系统性忽视。基于这些发现，MedCheck既可作为现有基准的诊断工具，也可作为一份可操作的指南，以促进在医疗领域采用更标准化、更可靠、更透明的方式评估人工智能。