Image anomaly detection (IAD) is an emerging and vital computer vision task in industrial manufacturing (IM). Recently many advanced algorithms have been published, but their performance deviates greatly. We realize that the lack of actual IM settings most probably hinders the development and usage of these methods in real-world applications. As far as we know, IAD methods are not evaluated systematically. As a result, this makes it difficult for researchers to analyze them because they are designed for different or special cases. To solve this problem, we first propose a uniform IM setting to assess how well these algorithms perform, which includes several aspects, i.e., various levels of supervision (unsupervised vs. semi-supervised), few-shot learning, continual learning, noisy labels, memory usage, and inference speed. Moreover, we skillfully build a comprehensive image anomaly detection benchmark (IM-IAD) that includes 16 algorithms on 7 mainstream datasets with uniform settings. Our extensive experiments (17,017 in total) provide in-depth insights for IAD algorithm redesign or selection under the IM setting. Next, the proposed benchmark IM-IAD gives challenges as well as directions for the future. To foster reproducibility and accessibility, the source code of IM-IAD is uploaded on the website, https://github.com/M-3LAB/IM-IAD.

翻译：图像异常检测（IAD）是工业制造（IM）领域一项新兴且至关重要的计算机视觉任务。近年来，许多先进算法已发布，但其性能差异显著。我们意识到，缺乏实际的工业制造场景很可能阻碍了这些方法在真实应用中的开发与使用。据我们所知，目前尚无系统性评估图像异常检测方法的手段。因此，由于这些方法针对不同或特殊场景设计，研究人员难以对其进行分析。为解决此问题，我们首先提出一个统一的工业制造场景来评估各算法的性能，该场景涵盖多个方面，包括不同监督级别（无监督与半监督）、小样本学习、持续学习、噪声标签、内存使用及推理速度。此外，我们精心构建了一个全面的图像异常检测基准（IM-IAD），包含7个主流数据集上的16种算法，并采用统一场景设置。我们的大量实验（总计17017次）为工业制造场景下的图像异常检测算法重新设计或选择提供了深入见解。随后，所提出的IM-IAD基准提出了未来挑战与方向。为促进可复现性与可访问性，IM-IAD的源代码已上传至网站：https://github.com/M-3LAB/IM-IAD。