Labeled data are critical to modern machine learning applications, but obtaining labels can be expensive. To mitigate this cost, machine learning methods, such as transfer learning, semi-supervised learning and active learning, aim to be label-efficient: achieving high predictive performance from relatively few labeled examples. While obtaining the best label-efficiency in practice often requires combinations of these techniques, existing benchmark and evaluation frameworks do not capture a concerted combination of all such techniques. This paper addresses this deficiency by introducing LabelBench, a new computationally-efficient framework for joint evaluation of multiple label-efficient learning techniques. As an application of LabelBench, we introduce a novel benchmark of state-of-the-art active learning methods in combination with semi-supervised learning for fine-tuning pretrained vision transformers. Our benchmark demonstrates better label-efficiencies than previously reported in active learning. LabelBench's modular codebase is open-sourced for the broader community to contribute label-efficient learning methods and benchmarks. The repository can be found at: https://github.com/EfficientTraining/LabelBench.

翻译：标注数据对现代机器学习应用至关重要，但获取标签成本高昂。为降低这一成本，机器学习方法（如迁移学习、半监督学习和主动学习）旨在实现标签高效性：即通过相对较少的标注示例获得高预测性能。尽管在实践中实现最佳标签高效性通常需要结合多种技术，但现有基准测试和评估框架尚未涵盖这些技术的协同组合。本文通过引入LabelBench填补了这一空白，这是一个新型计算高效的框架，用于联合评估多种标签高效学习技术。作为LabelBench的应用，我们提出了一个创新基准测试，将最先进的主动学习方法与半监督学习相结合，用于微调预训练视觉变换器。我们的基准测试展现了比此前主动学习报告更优的标签高效性。LabelBench的模块化代码库已开源，欢迎广大社区贡献标签高效学习方法与基准测试。资源库地址：https://github.com/EfficientTraining/LabelBench。