While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

翻译：尽管自动图像分析的重要性持续增长，但近期元研究揭示了算法验证方面的重大缺陷。性能度量标准对于有意义、客观且透明的性能评估以及所用自动算法的验证尤为关键，然而在针对特定图像分析任务使用具体度量标准时，其实际陷阱并未得到足够关注。这些陷阱通常与以下因素相关：(1) 忽视固有度量特性，例如在类别不平衡或目标结构较小情况下的表现行为；(2) 忽视固有数据集特性，例如测试案例的非独立性；(3) 忽视度量标准本应反映的实际生物医学领域关注点。本文作为一份动态更新的文献，旨在阐明图像分析领域常用性能度量标准的重要局限性。在此背景下，重点聚焦可表述为图像级分类、语义分割、实例分割或目标检测任务的生物医学图像分析问题。当前版本基于由全球60余家机构的图像分析专家组成的国际联盟开展的度量标准德尔菲法研究。