Recent advances in machine learning have led to increased deployment of black-box classifiers across a wide variety of applications. In many such situations there is a crucial need to assess the performance of these pre-trained models, for instance to ensure sufficient predictive accuracy, or that class probabilities are well-calibrated. Furthermore, since labeled data may be scarce or costly to collect, it is desirable for such assessment be performed in an efficient manner. In this paper, we introduce a Bayesian approach for model assessment that satisfies these desiderata. We develop inference strategies to quantify uncertainty for common assessment metrics (accuracy, misclassification cost, expected calibration error), and propose a framework for active assessment using this uncertainty to guide efficient selection of instances for labeling. We illustrate the benefits of our approach in experiments assessing the performance of modern neural classifiers (e.g., ResNet and BERT) on several standard image and text classification datasets.

翻译：最近在机器学习方面的进步导致在各种应用中更多地部署黑盒分类器,在许多情况下,迫切需要评估这些预先培训的模型的性能,例如确保充分的预测准确性,或对等级概率进行适当校准;此外,由于标签数据可能稀缺或收集费用高,因此宜以高效率的方式进行这种评估;在本文件中,我们采用贝叶斯式模型评估方法,满足了这些缺陷;我们制定推论战略,量化共同评估指标的不确定性(准确性、错误分类成本、预期校准错误),并提议一个利用这种不确定性进行积极评估的框架,以指导有效选择标签实例。我们举例说明了在几个标准图像和文本分类数据集上对现代神经分类器(例如ResNet和BERT)的性能进行实验的好处。