The goal for classification is to correctly assign labels to unseen samples. However, most methods misclassify samples with unseen labels and assign them to one of the known classes. Open-Set Classification (OSC) algorithms aim to maximize both closed and open-set recognition capabilities. Recent studies showed the utility of such algorithms on small-scale data sets, but limited experimentation makes it difficult to assess their performances in real-world problems. Here, we provide a comprehensive comparison of various OSC algorithms, including training-based (SoftMax, Garbage, EOS) and post-processing methods (Maximum SoftMax Scores, Maximum Logit Scores, OpenMax, EVM, PROSER), the latter are applied on features from the former. We perform our evaluation on three large-scale protocols that mimic real-world challenges, where we train on known and negative open-set samples, and test on known and unknown instances. Our results show that EOS helps to improve performance of almost all post-processing algorithms. Particularly, OpenMax and PROSER are able to exploit better-trained networks, demonstrating the utility of hybrid models. However, while most algorithms work well on negative test samples -- samples of open-set classes seen during training -- they tend to perform poorly when tested on samples of previously unseen unknown classes, especially in challenging conditions.

翻译：分类的目标是正确为未见样本分配标签。然而，大多数方法会对具有未见标签的样本进行错误分类，并将其归入已知类别之一。开放集分类（OSC）算法旨在同时最大化闭集与开放集识别能力。近期研究展示了此类算法在小规模数据集上的有效性，但有限的实验使其难以评估它们在实际问题中的性能。本文对多种OSC算法进行了全面比较，包括基于训练的方法（SoftMax、Garbage、EOS）和后处理方法（最大SoftMax分数、最大Logit分数、OpenMax、EVM、PROSER），后者应用于前者提取的特征之上。我们在三个模拟现实挑战的大规模协议上进行评估，使用已知类和负开放集样本进行训练，并在已知类与未知类实例上进行测试。实验结果表明，EOS有助于提升几乎所有后处理算法的性能。特别地，OpenMax和PROSER能够利用训练更充分的网络，证明了混合模型的有效性。然而，尽管大多数算法在负测试样本（训练期间见过的开放集类别样本）上表现良好，但在测试先前未见未知类别的样本时性能往往不佳，尤其在具有挑战性的条件下。