The ability to quickly and accurately identify covariate shift at test time is a critical and often overlooked component of safe machine learning systems deployed in high-risk domains. While methods exist for detecting when predictions should not be made on out-of-distribution test examples, identifying distributional level differences between training and test time can help determine when a model should be removed from the deployment setting and retrained. In this work, we define harmful covariate shift (HCS) as a change in distribution that may weaken the generalization of a predictive model. To detect HCS, we use the discordance between an ensemble of classifiers trained to agree on training data and disagree on test data. We derive a loss function for training this ensemble and show that the disagreement rate and entropy represent powerful discriminative statistics for HCS. Empirically, we demonstrate the ability of our method to detect harmful covariate shift with statistical certainty on a variety of high-dimensional datasets. Across numerous domains and modalities, we show state-of-the-art performance compared to existing methods, particularly when the number of observed test samples is small.

翻译：在测试阶段快速且准确地识别协变量偏移的能力是高风险领域中安全机器学习系统的一个关键且常被忽视的组成部分。虽然现有方法能够检测何时不应在分布外测试样本上做出预测，但识别训练与测试阶段之间的分布级差异，有助于确定模型何时需要从部署环境中移除并进行重新训练。在本文中，我们将有害协变量偏移（HCS）定义为可能削弱预测模型泛化能力的分布变化。为了检测HCS，我们利用了在训练数据上达成一致但在测试数据上产生分歧的集成分类器之间的不一致性。我们推导了用于训练该集成模型的损失函数，并证明分歧率和熵是检测HCS的有力判别统计量。通过实验，我们在多种高维数据集上展示了该方法以统计确定性检测有害协变量偏移的能力。在众多领域和模态中，我们证明了相较于现有方法，该方法在观测测试样本数量较少时尤其能够达到最先进的性能。