Recent advances in visual anomaly detection research have seen AUROC and AUPRO scores on public benchmark datasets such as MVTec and VisA converge towards perfect recall, giving the impression that these benchmarks are near-solved. However, high AUROC and AUPRO scores do not always reflect qualitative performance, which limits the validity of these metrics in real-world applications. We argue that the artificial ceiling imposed by the lack of an adequate evaluation metric restrains progression of the field, and it is crucial that we revisit the evaluation metrics used to rate our algorithms. In response, we introduce Per-IMage Overlap (PIMO), a novel metric that addresses the shortcomings of AUROC and AUPRO. PIMO retains the recall-based nature of the existing metrics but introduces two distinctions: the assignment of curves (and respective area under the curve) is per-image, and its X-axis relies solely on normal images. Measuring recall per image simplifies instance score indexing and is more robust to noisy annotations. As we show, it also accelerates computation and enables the usage of statistical tests to compare models. By imposing low tolerance for false positives on normal images, PIMO provides an enhanced model validation procedure and highlights performance variations across datasets. Our experiments demonstrate that PIMO offers practical advantages and nuanced performance insights that redefine anomaly detection benchmarks -- notably challenging the perception that MVTec AD and VisA datasets have been solved by contemporary models. Available on GitHub: https://github.com/jpcbertoldo/aupimo.

翻译：近期视觉异常检测研究的进展表明，在MVTec和VisA等公开基准数据集上，AUROC和AUPRO得分已趋近完美召回率，给人造成这些基准问题几乎已被解决的印象。然而，高AUROC和AUPRO得分并不总能反映定性性能，这限制了这些指标在实际应用中的有效性。我们认为，缺乏适当评估指标所造成的人为天花板限制了该领域的发展，重新审视用于评估算法的指标至关重要。为此，我们提出逐图像重叠度（PIMO）这一新型指标，以解决AUROC和AUPRO的缺陷。PIMO保留了现有指标基于召回率的特性，但引入了两个区别：曲线（及其相应曲线下面积）的分配是按图像进行的，且其X轴仅依赖于正常图像。逐图像测量召回率简化了实例得分的索引，并对噪声标注更具鲁棒性。如我们所示，该方法还能加速计算，并支持使用统计检验来比较模型。通过对正常图像施加低误报容忍度，PIMO提供了增强的模型验证程序，并凸显了跨数据集的性能差异。实验表明，PIMO提供了实用优势与细微的性能洞察，重新定义了异常检测基准——尤其挑战了当代模型已解决MVTec AD和VisA数据集的认知。GitHub开源地址：https://github.com/jpcbertoldo/aupimo。