Recent advances in visual anomaly detection research have seen AUROC and AUPRO scores on public benchmark datasets such as MVTec and VisA converge towards perfect recall, giving the impression that these benchmarks are near-solved. However, high AUROC and AUPRO scores do not always reflect qualitative performance, which limits the validity of these metrics in real-world applications. We argue that the artificial ceiling imposed by the lack of an adequate evaluation metric restrains progression of the field, and it is crucial that we revisit the evaluation metrics used to rate our algorithms. In response, we introduce Per-IMage Overlap (PIMO), a novel metric that addresses the shortcomings of AUROC and AUPRO. PIMO retains the recall-based nature of the existing metrics but introduces two distinctions: the assignment of curves (and respective area under the curve) is per-image, and its X-axis relies solely on normal images. Measuring recall per image simplifies instance score indexing and is more robust to noisy annotations. As we show, it also accelerates computation and enables the usage of statistical tests to compare models. By imposing low tolerance for false positives on normal images, PIMO provides an enhanced model validation procedure and highlights performance variations across datasets. Our experiments demonstrate that PIMO offers practical advantages and nuanced performance insights that redefine anomaly detection benchmarks -- notably challenging the perception that MVTec AD and VisA datasets have been solved by contemporary models. Available on GitHub: https://github.com/jpcbertoldo/aupimo.

翻译：近期视觉异常检测研究的进展使得MVTec、VisA等公开基准数据集上的AUROC和AUPRO分数趋近完美召回，给人以这些基准任务已近解决的印象。然而，高AUROC和AUPRO分数并不总能反映定性性能，这限制了这些指标在真实应用中的有效性。我们认为，缺乏充分评估指标所造成的人为天花板制约了该领域的发展，因此重新审视用于评估算法的指标至关重要。为此，我们提出"每图像重叠度"（Per-IMage Overlap, PIMO）这一新型指标，以应对AUROC和AUPRO的缺陷。PIMO保留了现有指标基于召回率的特性，但引入两点区别：曲线（及其对应曲线下面积）按图像分配，且其X轴仅依赖正常图像。逐图像测量召回率简化了实例分数索引，并对噪声标注更具鲁棒性。如我们所示，这还能加速计算并支持使用统计检验比较模型。通过对正常图像施加低误报容错，PIMO提供了增强的模型验证流程，并揭示了跨数据集的性能差异。实验表明，PIMO提供了实用的优势与细微的性能洞察，从根本上重新定义了异常检测基准——尤其挑战了MVTec AD与VisA数据集已被当代模型解决的普遍认知。代码已开源至GitHub：https://github.com/jpcbertoldo/aupimo。