Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. One simple deferral rule employs the confidence of the current classifier, e.g., based on the maximum predicted softmax probability. Despite being oblivious to the structure of the cascade -- e.g., not modelling the errors of downstream models -- such confidence-based deferral often works remarkably well in practice. In this paper, we seek to better understand the conditions under which confidence-based deferral may fail, and when alternate deferral strategies can perform better. We first present a theoretical characterisation of the optimal deferral rule, which precisely characterises settings under which confidence-based deferral may suffer. We then study post-hoc deferral mechanisms, and demonstrate they can significantly improve upon confidence-based deferral in settings where (i) downstream models are specialists that only work well on a subset of inputs, (ii) samples are subject to label noise, and (iii) there is distribution shift between the train and test set.

翻译：级联是一种经典策略，通过对样本依次调用分类器序列，实现推理成本的自适应变化。推迟规则决定是否调用序列中的下一个分类器或终止预测。一种简单的推迟规则基于当前分类器的置信度（例如最大预测softmax概率）。尽管这种基于置信度的推迟策略不依赖于级联结构（例如不建模下游模型的错误），但在实践中往往表现优异。本文旨在深入理解基于置信度的推迟策略可能失效的条件，以及替代推迟策略何时能取得更优表现。我们首先对最优推迟规则进行理论刻画，精确描述基于置信度的推迟策略可能失效的场景。随后研究事后推迟机制，并证明在下游模型为仅在部分输入上表现良好的专家、样本存在标签噪声、训练集与测试集存在分布偏移这三种场景下，该机制可显著优于基于置信度的推迟策略。