Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterizing the fit of the model to the underlying conditional law of labels given the features vector ($Y|X$), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law $Y|X$, and treats that as a black box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form \[ H_0: \mathbb{E}\Big[D_f\Big({\sf Bern}(\eta(X))\|{\sf Bern}(\hat{\eta}(X))\Big)\Big]\leq \tau\,, \] where $D_f$ represents an $f$-divergence function, and $\eta(x)$, $\hat{\eta}(x)$ respectively denote the true and an estimate likelihood for a feature vector $x$ admitting a positive label. We propose a novel test, called \grasp for testing $H_0$, which works in finite sample settings, no matter the features (distribution-free). We also propose model-X \grasp designed for model-X settings where the joint distribution of the features vector is known. Model-X \grasp uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.

翻译：分类器的性能通常以测试数据的平均准确度来衡量。 尽管这是标准尺度, 但平均准确度无法将模型与特性矢量( Y ⁇ X$) 所显示的标签基本有条件法的匹配性格( y ⁇ X$ ), 例如由于模型区分不当、 安装过重、 高维度等原因。 在本文中, 我们考虑评估通用二进制分类器是否适合的基本问题。 我们的框架没有对条件法作任何参数假设 $Y ⁇ X$\ x$\Big\Big\Big\\Big\leq\\ tau\\, 并且将它作为黑盒或甲模型模型模型模型模型模型模型模型模型, 只能通过查询获得。 我们将“ 良好” 评估特性作为表格[ H_0:\ mathb{E\Big[D_Big_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_ 的容忍性假设性能, 用于新矢量度测试( We_x) 的模型- deal- deal- deal_al_ tral_ tral_ a exal_ a exal_ a exexexexex test ex testal_ a ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex exestal ex ex ex ex ex ex ex ex exal exal exestal exestaleveralal exal exal exal ex exal exal ex ex exeveral a a a a a a ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex ex