Decision support systems are designed to assist human experts in classification tasks by providing conformal prediction sets derived from a pre-trained model. This human-AI collaboration has demonstrated enhanced classification performance compared to using either the model or the expert independently. In this study, we focus on the selection of instance-specific experts from a pool of multiple human experts, contrasting it with existing research that typically focuses on single-expert scenarios. We characterize the conditions under which multiple experts can benefit from the conformal sets. With the insight that only certain experts may be relevant for each instance, we explore the problem of subset selection and introduce a greedy algorithm that utilizes conformal sets to identify the subset of expert predictions that will be used in classifying an instance. This approach is shown to yield better performance compared to naive methods for human subset selection. Based on real expert predictions from the CIFAR-10H and ImageNet-16H datasets, our simulation study indicates that our proposed greedy algorithm achieves near-optimal subsets, resulting in improved classification performance among multiple experts.

翻译：决策支持系统旨在通过提供源自预训练模型的置信预测集，协助人类专家完成分类任务。相较于单独使用模型或专家，这种人机协作已展现出更优越的分类性能。本研究聚焦于从多专家池中为特定实例选择专家，与现有通常关注单专家场景的研究形成对比。我们系统阐述了多专家能从置信集中获益的条件。基于"仅部分专家对每个实例具有相关性"的洞见，我们深入探讨了专家子集选择问题，并提出一种利用置信集的贪心算法，以识别用于实例分类的专家预测子集。相较于简单的人工子集选择方法，该策略展现出更优的性能。基于CIFAR-10H和ImageNet-16H数据集的真实专家预测数据，我们的仿真研究表明：所提出的贪心算法能获得接近最优的专家子集，从而显著提升多专家场景下的分类性能。