We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors over time. Our solution extends this framework to work in the absence of labels, by employing a proxy for the true error. This proxy is derived using the predictions of a trained error estimator. Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time.

翻译：本文提出一种新颖方法，用于在持续生产环境中检测对机器学习模型性能产生负面影响的分布偏移，该方法无需访问真实数据标签。本研究基于Podkopaev和Ramdas [2022]的工作——该工作解决了可通过标签追踪模型时序误差的场景。我们的解决方案通过采用真实误差的代理指标，将该框架扩展至无标签场景。该代理指标通过训练好的误差估计器的预测结果推导得出。实验表明，在协变量偏移、标签偏移以及跨地域和时间的自然偏移等多种分布偏移情况下，本方法具有较高的检测功效与误报控制能力。