The ability to quickly and accurately identify covariate shift at test time is a critical and often overlooked component of safe machine learning systems deployed in high-risk domains. While methods exist for detecting when predictions should not be made on out-of-distribution test examples, identifying distributional level differences between training and test time can help determine when a model should be removed from the deployment setting and retrained. In this work, we define harmful covariate shift (HCS) as a change in distribution that may weaken the generalization of a predictive model. To detect HCS, we use the discordance between an ensemble of classifiers trained to agree on training data and disagree on test data. We derive a loss function for training this ensemble and show that the disagreement rate and entropy represent powerful discriminative statistics for HCS. Empirically, we demonstrate the ability of our method to detect harmful covariate shift with statistical certainty on a variety of high-dimensional datasets. Across numerous domains and modalities, we show state-of-the-art performance compared to existing methods, particularly when the number of observed test samples is small.

翻译：在测试时快速准确识别协变量偏移的能力，是高风险领域部署安全机器学习系统中常被忽视的关键环节。现有方法虽能检测何时不应在分布外测试样本上进行预测，但识别训练与测试时分布层面的差异，有助于判断模型何时应从部署环境中移除并重新训练。本文将危害性协变量偏移（HCS）定义为可能削弱预测模型泛化能力的分布变化。为检测HCS，我们利用一组在训练数据上趋于一致、在测试数据上产生分歧的集成分类器之间的不一致性。我们推导出训练该集成模型的损失函数，并证明分歧率与熵是检测HCS的强大判别统计量。实验表明，在多种高维数据集上，我们的方法能够以统计确定性检测危害性协变量偏移。在众多领域与模态中，我们展示了相比现有方法更优的性能，尤其在观测测试样本数量较少的情况下表现突出。