Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. These predictions can then be deferred to humans for further evaluation. As an everlasting challenge for machine learning, in many real-world scenarios, the distribution of test data is different from the training data. This results in more inaccurate predictions, and often increased dependence on humans, which can be difficult and expensive. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. Selective prediction and active learning have been approached from different angles, with the connection between them missing. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain while increasing accuracy and coverage. For this new paradigm, we propose a simple yet effective approach, ASPEST, that utilizes ensembles of model snapshots with self-training with their aggregated outputs as pseudo labels. Extensive experiments on numerous image, text and structured datasets, which suffer from domain shifts, demonstrate that ASPEST can significantly outperform prior work on selective prediction and active learning (e.g. on the MNIST$\to$SVHN benchmark with the labeling budget of 100, ASPEST improves the AUACC metric from 79.36% to 88.84%) and achieves more optimal utilization of humans in the loop.

翻译：选择性预测旨在学习一个可靠的模型，使其在面对不确定时放弃预测，并将这些预测转交给人类进行进一步评估。作为机器学习领域的一项长期挑战，在许多现实场景中，测试数据的分布与训练数据不同，这会导致更多不准确的预测，并往往增加对人类依赖的难度和成本。主动学习则通过查询最具信息量的样本来降低整体标注成本，从而减少对人类依赖。选择性预测与主动学习从不同角度被探索，但两者之间的联系尚未建立。本文提出了一种新的学习范式，即主动选择性预测，旨在从发生偏移的目标领域中查询更具信息量的样本，同时提升准确率和覆盖率。针对这一新范式，我们提出了一种简单而有效的方法ASPEST，该方法利用模型快照的集成，并结合其聚合输出作为伪标签进行自训练。在多个涉及域偏移的图像、文本和结构化数据集上进行的大量实验表明，ASPEST可以显著优于此前选择性预测和主动学习的研究（例如，在标注预算为100的MNIST→SVHN基准测试上，ASPEST将AUACC指标从79.36%提升至88.84%），并实现了更优的人机协同利用。