We consider the problem of evaluating black-box multi-class classifiers. In the standard setup, we observe class labels $Y\in \{0,1,\ldots,M-1\}$ generated according to the conditional distribution $ Y|X \sim \text{ Multinom}\big(η(X)\big), $ where $X$ denotes the features and $η$ maps from the feature space to the $(M-1)$-dimensional simplex. A black-box classifier is an estimate $\hatη$ for which we make no assumptions about the training algorithm. Given holdout data, our goal is to evaluate the performance of the classifier $\hatη$. Recent work suggests treating this as a goodness-of-fit problem by testing the hypothesis $H_0: ρ((X,Y),(X',Y')) \le δ$, where $ρ$ is some metric between two distributions, and $(X',Y')\sim P_X\times \text{ Multinom}(\hatη(X))$. Combining ideas from algorithmic fairness, Neyman-Pearson lemma, and conformal p-values, we propose a new methodology for this testing problem. The key idea is to generate a second sample $(X',Y') \sim P_X \times \text{ Multinom}\big(\hatη(X)\big)$ allowing us to reduce the task to two-sample conditional distribution testing. Using part of the data, we train an auxiliary binary classifier called a distinguisher to attempt to distinguish between the two samples. The distinguisher's ability to differentiate samples, measured using a rank-sum statistic, is then used to assess the difference between $\hatη$ and $η$ . Using techniques from cross-validation central limit theorems, we derive an asymptotically rigorous test under suitable stability conditions of the distinguisher.

翻译：我们考虑评估黑盒多类分类器的问题。在标准设定中，观测到根据条件分布 $ Y|X \sim \text{ Multinom}\big(η(X)\big) $ 生成的类别标签 $Y\in \{0,1,\ldots,M-1\}$ ，其中 $X$ 表示特征，$η$ 是从特征空间到 $(M-1)$ 维单形的映射。黑盒分类器是对 $\hatη$ 的估计，且不对其训练算法做任何假设。给定保留数据集，我们的目标是评估分类器 $\hatη$ 的性能。近期研究建议将此问题视为拟合优度检验，通过验证假设 $H_0: ρ((X,Y),(X',Y')) \le δ$ ，其中 $ρ$ 为两个分布之间的某种度量，且 $(X',Y')\sim P_X\times \text{ Multinom}(\hatη(X))$ 。结合算法公平性、Neyman-Pearson引理和保形p值的思想，我们针对这一检验问题提出了一种新方法。关键思路是生成第二个样本 $(X',Y') \sim P_X \times \text{ Multinom}\big(\hatη(X)\big)$ ，从而将任务简化为条件分布的双样本检验。利用部分数据，我们训练一个称为区分器的辅助二分类器，以尝试区分两个样本。区分器区分样本的能力通过秩和统计量进行度量，进而用于评估 $\hatη$ 与 $η$ 之间的差异。借助交叉验证中心极限定理的技术，我们在区分器满足适当稳定性条件下推导出了渐近严格的检验方法。