Automated decision support systems promise to help human experts solve multiclass classification tasks more efficiently and accurately. However, existing systems typically require experts to understand when to cede agency to the system or when to exercise their own agency. Otherwise, the experts may be better off solving the classification tasks on their own. In this work, we develop an automated decision support system that, by design, does not require experts to understand when to trust the system to improve performance. Rather than providing (single) label predictions and letting experts decide when to trust these predictions, our system provides sets of label predictions constructed using conformal prediction$\unicode{x2014}$prediction sets$\unicode{x2014}$and forcefully asks experts to predict labels from these sets. By using conformal prediction, our system can precisely trade-off the probability that the true label is not in the prediction set, which determines how frequently our system will mislead the experts, and the size of the prediction set, which determines the difficulty of the classification task the experts need to solve using our system. In addition, we develop an efficient and near-optimal search method to find the conformal predictor under which the experts benefit the most from using our system. Simulation experiments using synthetic and real expert predictions demonstrate that our system may help experts make more accurate predictions and is robust to the accuracy of the classifier the conformal predictor relies on.

翻译：自动化决策支持系统有望帮助人类专家更高效、更准确地解决多类分类任务。然而，现有系统通常要求专家理解何时应将决策权交给系统，或何时自行行使判断权。否则，专家独自处理分类任务可能效果更佳。本研究开发了一种无需专家判断何时信任系统即可提升性能的自动化决策支持系统。该系统不提供单一标签预测供专家决定是否采纳，而是生成基于保形预测构建的标签预测集（即预测集），并强制要求专家从这些集合中预测标签。通过使用保形预测，系统能够精确平衡两类指标：真实标签不在预测集中的概率（决定系统误导专家的频率）以及预测集的大小（决定专家使用系统时需解决的分类任务难度）。此外，我们提出了一种高效且近似最优的搜索方法，用于找到最有利于专家使用系统的保形预测器。基于合成数据与真实专家预测的仿真实验表明，该系统可帮助专家做出更准确的预测，且对保形预测器所依赖的分类器精度具有鲁棒性。