Classifier models are prevalent in natural language processing (NLP), often with high accuracy. Yet in real world settings, human-in-the-loop systems can foster trust in model outputs and even higher performance. Selective Prediction (SP) methods determine when to adopt a classifier's output versus defer to a human. Previous SP approaches have addressed how to improve softmax as a measure of model confidence, or have developed separate confidence estimators. One previous method involves learning a deferral model based on engineered features. We introduce a novel joint-training approach that simultaneously optimizes learned representations used by the classifier module and a learned deferral policy. Our results on four classification tasks demonstrate that joint training not only leads to better SP outcomes over two strong baselines, but also improves the performance of both modules.

翻译：分类器模型在自然语言处理（NLP）领域应用广泛，通常具有较高的准确性。然而，在现实场景中，人机协同系统能够增强对模型输出的信任，甚至实现更高的性能。选择性预测（SP）方法旨在决定何时采纳分类器的输出，何时交由人类处理。先前的研究方法或致力于改进将softmax作为模型置信度度量的方式，或开发了独立的置信度估计器。其中一种方法涉及基于人工设计特征学习一个延迟模型。本文提出了一种新颖的联合训练方法，该方法同时优化分类器模块使用的学习表示和一个学习得到的延迟策略。我们在四个分类任务上的实验结果表明，联合训练不仅相较于两个强基线模型带来了更好的选择性预测结果，而且提升了两个模块各自的性能。