Image anomaly detection (IAD) is an emerging and vital computer vision task in industrial manufacturing (IM). Recently many advanced algorithms have been published, but their performance deviates greatly. We realize that the lack of actual IM settings most probably hinders the development and usage of these methods in real-world applications. As far as we know, IAD methods are not evaluated systematically. As a result, this makes it difficult for researchers to analyze them because they are designed for different or special cases. To solve this problem, we first propose a uniform IM setting to assess how well these algorithms perform, which includes several aspects, i.e., various levels of supervision (unsupervised vs. semi-supervised), few-shot learning, continual learning, noisy labels, memory usage, and inference speed. Moreover, we skillfully build a comprehensive image anomaly detection benchmark (IM-IAD) that includes 16 algorithms on 7 mainstream datasets with uniform settings. Our extensive experiments (17,017 in total) provide in-depth insights for IAD algorithm redesign or selection under the IM setting. Next, the proposed benchmark IM-IAD gives challenges as well as directions for the future. To foster reproducibility and accessibility, the source code of IM-IAD is uploaded on the website, https://github.com/M-3LAB/IM-IAD.

翻译：摘要：图像异常检测（IAD）是工业制造（IM）中一项新兴且至关重要的计算机视觉任务。近年来，尽管已发表了许多先进算法，但它们的性能差异显著。我们认识到，实际工业制造场景的缺失很可能阻碍了这些方法在真实应用中的发展和使用。据我们所知，现有IAD方法尚未经过系统性评估。因此，研究人员难以对它们进行分析，因为这些方法通常针对不同或特殊场景设计。为解决这一问题，我们首先提出了统一的工业制造评估设置，从多个维度衡量算法性能，包括：不同监督级别（无监督 vs. 半监督）、小样本学习、持续学习、噪声标签、内存使用率及推理速度。此外，我们巧妙构建了全面的图像异常检测基准（IM-IAD），在统一设置下涵盖7个主流数据集上的16种算法。通过大量实验（总计17,017次），我们为工业制造场景下的IAD算法重新设计或选择提供了深入见解。随后，所提出的IM-IAD基准为未来研究提出了挑战与方向。为促进可复现性与可访问性，IM-IAD的源代码已上传至网站：https://github.com/M-3LAB/IM-IAD。