Recent advances in visual anomaly detection research have seen AUROC and AUPRO scores on public benchmark datasets such as MVTec and VisA converge towards perfect recall, giving the impression that these benchmarks are near-solved. However, high AUROC and AUPRO scores do not always reflect qualitative performance, which limits the validity of these metrics in real-world applications. We argue that the artificial ceiling imposed by the lack of an adequate evaluation metric restrains progression of the field, and it is crucial that we revisit the evaluation metrics used to rate our algorithms. In response, we introduce Per-IMage Overlap (PIMO), a novel metric that addresses the shortcomings of AUROC and AUPRO. PIMO retains the recall-based nature of the existing metrics but introduces two distinctions: the assignment of curves (and respective area under the curve) is per-image, and its X-axis relies solely on normal images. Measuring recall per image simplifies instance score indexing and is more robust to noisy annotations. As we show, it also accelerates computation and enables the usage of statistical tests to compare models. By imposing low tolerance for false positives on normal images, PIMO provides an enhanced model validation procedure and highlights performance variations across datasets. Our experiments demonstrate that PIMO offers practical advantages and nuanced performance insights that redefine anomaly detection benchmarks -- notably challenging the perception that MVTec AD and VisA datasets have been solved by contemporary models. Available on GitHub: https://github.com/jpcbertoldo/aupimo.

翻译：近期视觉异常检测研究的进展表明，在MVTec、VisA等公开基准数据集上，AUROC和AUPRO分数已趋近于完美召回率，令人产生这些基准问题已接近解决的印象。然而，高AUROC和AUPRO分数并不总能反映定性性能，这在现实应用中限制了这些指标的有效性。我们认为，因缺乏充分评估指标而产生的人为上限制约了该领域的发展，重新审视用于评定算法性能的评估指标至关重要。为此，我们提出逐图像重叠（Per-IMage Overlap, PIMO）这一新型指标，以弥补AUROC和AUPRO的不足。PIMO保留了现有指标基于召回率的本质，但引入两点区别：曲线（及其对应曲线下面积）的分配按图像进行，且其X轴仅依赖正常图像。逐图像测量召回率可简化实例分数索引，并对噪声标注更具鲁棒性。如我们所示，该方法还能加速计算，并支持利用统计检验比较模型。通过对正常图像施加低假阳性容错，PIMO增强了模型验证流程，凸显了跨数据集的性能差异。实验表明，PIMO提供了实用优势与细微的性能洞察，重新定义了异常检测基准——尤其挑战了MVTec AD和VisA数据集已被当代模型解决的认知。代码开源于GitHub：https://github.com/jpcbertoldo/aupimo。