Complementary-label learning (CLL) is a weakly-supervised learning paradigm that aims to train a multi-class classifier using only complementary labels, which indicate classes to which an instance does not belong. Despite numerous algorithmic proposals for CLL, their practical performance remains unclear for two reasons. Firstly, these algorithms often rely on assumptions about the generation of complementary labels. Secondly, their evaluation has been limited to synthetic datasets. To gain insights into the real-world performance of CLL algorithms, we developed a protocol to collect complementary labels annotated by human annotators. This effort resulted in the creation of two datasets, CLCIFAR10 and CLCIFAR20, derived from CIFAR10 and CIFAR100, respectively. These datasets, publicly released at https://github.com/ntucllab/complementary_cifar, represent the very first real-world CLL datasets. Through extensive benchmark experiments, we discovered a notable decline in performance when transitioning from synthetic datasets to real-world datasets. We conducted a dataset-level ablation study to investigate the key factors contributing to this decline. Our analyses highlighted annotation noise as the most influential factor present in the real-world datasets. Additionally, the biased nature of human-annotated complementary labels was found to make certain CLL algorithms more susceptible to overfitting. These findings suggest the community to spend more research effort on developing CLL algorithms that are robust to noisy and biased complementary-label distributions.

翻译：补标签学习（CLL）是一种弱监督学习范式，旨在仅利用补标签（即指示实例不属于的类别）来训练多类分类器。尽管已有大量关于CLL的算法研究，但其实际性能仍因两个原因而不明确：首先，这些算法通常依赖于关于补标签生成的假设；其次，其评估仅限于合成数据集。为深入了解CLL算法在真实场景中的性能，我们设计了一种协议来收集由人工标注者标注的补标签。这一工作最终创建了两个数据集——CLCIFAR10和CLCIFAR20，它们分别源自CIFAR10和CIFAR100。这些数据集已在https://github.com/ntucllab/complementary_cifar上公开发布，代表了首个真实世界的CLL数据集。通过广泛的基准实验，我们发现从合成数据集迁移到真实数据集时性能显著下降。我们进行了数据集级别的消融研究，以探究导致性能下降的关键因素。分析表明，标注噪声是真实数据集中的最主要影响因素。此外，人工标注补标签的偏倚性质使某些CLL算法更容易过拟合。这些发现建议学界投入更多研究精力，开发对噪声和偏倚补标签分布具有鲁棒性的CLL算法。