Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.

翻译：给定一个预训练分类器及多位人类专家，我们研究在线分类任务：模型预测可免费获取，但查询专家意见需付出成本。在这一实用但尚未充分探索的场景中，无法获取真实标签；相反，预测目标被定义为所有专家的共识投票结果。考虑到查询完整共识的高昂成本，我们提出一个通用的在线贝叶斯共识估计框架，利用多元超几何分布的性质。基于该框架，我们推出一系列方法，通过生成关于专家与模型信念的后验分布，从部分反馈中动态估计专家共识。该后验分析可在查询成本与分类性能之间建立可解释的权衡。我们在CIFAR-10H与ImageNet-16H两个大规模众包数据集上，验证了该框架相对于多种基线方法的有效性。