An open scientific challenge is how to classify events with reliable measures of uncertainty, when we have a mechanistic model of the data-generating process but the distribution over both labels and latent nuisance parameters is different between train and target data. We refer to this type of distributional shift as generalized label shift (GLS). Direct classification using observed data $\mathbf{X}$ as covariates leads to biased predictions and invalid uncertainty estimates of labels $Y$. We overcome these biases by proposing a new method for robust uncertainty quantification that casts classification as a hypothesis testing problem under nuisance parameters. The key idea is to estimate the classifier's receiver operating characteristic (ROC) across the entire nuisance parameter space, which allows us to devise cutoffs that are invariant under GLS. Our method effectively endows a pre-trained classifier with domain adaptation capabilities and returns valid prediction sets while maintaining high power. We demonstrate its performance on two challenging scientific problems in biology and astroparticle physics with data from realistic mechanistic models.

翻译：一个开放的科学挑战是：当拥有数据生成过程的机制模型，但训练数据与目标数据中标签和潜在干扰参数的分布不同时，如何以可靠的度量对事件进行分类并评估不确定性。我们将这种分布偏移称为广义标签偏移（GLS）。直接使用观测数据$\mathbf{X}$作为协变量进行分类会导致有偏的预测以及标签$Y$的不确定性估计失效。我们通过提出一种新的鲁棒不确定性量化方法来克服这些偏差，该方法将分类视为干扰参数下的假设检验问题。其核心思想是在整个干扰参数空间中估计分类器的受试者工作特征曲线（ROC），从而设计出在GLS下保持不变的截断阈值。我们的方法有效赋予预训练分类器领域适应能力，在保持高统计功效的同时返回有效的预测集。我们通过来自现实机制模型的数据，在生物学和天体粒子物理学两个具有挑战性的科学问题上展示了其性能。