Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

翻译：验证指标对于可靠追踪科学进展以及弥合人工智能（AI）研究与其实际应用之间的当前鸿沟至关重要。然而，越来越多的证据表明，尤其在图像分析领域，指标的选择往往与底层研究问题不相匹配。这可能是由于指标相关知识缺乏可及性：虽然考虑验证指标的各自优势、劣势和局限性是做出明智选择的先决条件，但相关知识目前分散且难以被个别研究者获取。基于由多学科专家联盟进行的多阶段德尔菲法流程以及广泛的社区反馈，本研究首次提供了关于图像分析验证中与指标相关陷阱的可靠且全面的通用信息入口。虽然聚焦于生物医学图像分析，但具有向其他领域迁移的潜力，所涉及陷阱的应用领域具有普遍性，并根据新创建的、与领域无关的分类法进行分类。为便于理解，每个陷阱均配有图示和具体示例。作为可供各级别研究者获取的结构化信息体系，本研究加深了全球对图像分析验证中这一关键主题的理解。