Out-of-distribution (OOD) detection is the problem of identifying inputs which are unrelated to the in-distribution task. The OOD detection performance when the in-distribution (ID) is ImageNet-1K is commonly being tested on a small range of test OOD datasets. We find that most of the currently used test OOD datasets, including datasets from the open set recognition (OSR) literature, have severe issues: In some cases more than 50$\%$ of the dataset contains objects belonging to one of the ID classes. These erroneous samples heavily distort the evaluation of OOD detectors. As a solution, we introduce with NINCO a novel test OOD dataset, each sample checked to be ID free, which with its fine-grained range of OOD classes allows for a detailed analysis of an OOD detector's strengths and failure modes, particularly when paired with a number of synthetic "OOD unit-tests". We provide detailed evaluations across a large set of architectures and OOD detection methods on NINCO and the unit-tests, revealing new insights about model weaknesses and the effects of pretraining on OOD detection performance. We provide code and data at https://github.com/j-cb/NINCO.

翻译：分布外（OOD）检测是指识别与内分布任务无关的输入问题。当内分布（ID）为ImageNet-1K时，OOD检测性能通常在一小部分测试OOD数据集上进行评测。我们发现，当前大多数使用的测试OOD数据集（包括来自开放集识别（OSR）文献的数据集）存在严重问题：部分数据集中超过50%的样本包含属于某个ID类别的物体。这些错误样本严重扭曲了OOD检测器的评估。为此，我们提出NINCO这一新颖的测试OOD数据集，其中每个样本均经检验不含ID类别。该数据集凭借其细粒度的OOD类别范围，可对OOD检测器的优势与失败模式进行详细分析，尤其当与一系列合成的“OOD单元测试”结合使用时效果更佳。我们在NINCO及单元测试上对大量架构和OOD检测方法进行了详细评估，揭示了关于模型弱点及预训练对OOD检测性能影响的新见解。我们在https://github.com/j-cb/NINCO提供代码与数据。