A growing body of work studies Blindspot Discovery Methods ("BDM"s): methods that use an image embedding to find semantically meaningful (i.e., united by a human-understandable concept) subsets of the data where an image classifier performs significantly worse. Motivated by observed gaps in prior work, we introduce a new framework for evaluating BDMs, SpotCheck, that uses synthetic image datasets to train models with known blindspots and a new BDM, PlaneSpot, that uses a 2D image representation. We use SpotCheck to run controlled experiments that identify factors that influence BDM performance (e.g., the number of blindspots in a model, or features used to define the blindspot) and show that PlaneSpot is competitive with and in many cases outperforms existing BDMs. Importantly, we validate these findings by designing additional experiments that use real image data from MS-COCO, a large image benchmark dataset. Our findings suggest several promising directions for future work on BDM design and evaluation. Overall, we hope that the methodology and analyses presented in this work will help facilitate a more rigorous science of blindspot discovery.

翻译：日益增多的研究关注盲点发现方法（BDMs）：这些方法利用图像嵌入来识别数据中语义上有意义（即由人类可理解的概念统一）的子集，在这些子集中图像分类器的表现显著较差。受先前工作中观察到空白的启发，我们引入了一个新的BDM评估框架SpotCheck，该框架使用合成图像数据集训练具有已知盲点的模型，并提出了一个新的BDM——PlaneSpot，它利用二维图像表示。我们利用SpotCheck进行受控实验，确定了影响BDM性能的因素（例如模型中盲点的数量或用于定义盲点的特征），并表明PlaneSpot与现有BDM相比具有竞争力，且在多数情况下性能更优。重要的是，我们通过设计额外实验验证了这些发现，这些实验使用了来自大型图像基准数据集MS-COCO的真实图像数据。我们的研究结果为未来BDM设计与评估的几个有前景的方向提供了启示。总体而言，我们希望本文提出的方法论和分析有助于推动更严谨的盲点发现科学的发展。