Monitoring for changes in a predictive relationship represented by a fitted supervised learning model (i.e., concept drift detection) is a widespread problem in modern data-driven applications. A general and powerful Fisher score-based concept drift approach was recently proposed, in which detecting concept drift reduces to detecting changes in the mean of the model's score vector using a multivariate exponentially weighted moving average (MEWMA). To implement the approach, the initial data must be split into two subsets. The first subset serves as the training sample to which the model is fit, and the second subset serves as an out-of-sample test set from which the MEWMA control limit (CL) is determined. In this paper, we retain the same score-based MEWMA monitoring statistic as the existing method and focus instead on improving the computation of the control limit. We develop a novel nested bootstrap procedure for calibrating the CL that allows the entire initial sample to be used for model fitting, thereby yielding a more accurate baseline model while eliminating the need for a large holdout set. We show that a standard nested bootstrap substantially underestimates the variability of the monitoring statistic and develop a 0.632-like correction that appropriately accounts for this. We demonstrate the advantages with numerical examples.

翻译：在现代数据驱动应用中，监测由拟合监督学习模型表示的预测关系的变化（即概念漂移检测）是一个普遍存在的问题。最近提出了一种通用且强大的基于Fisher得分的概念漂移检测方法，该方法通过多元指数加权移动平均（MEWMA）检测模型得分向量均值的变化来实现概念漂移检测。为实施该方法，初始数据必须被划分为两个子集：第一个子集作为训练样本用于拟合模型，第二个子集作为样本外测试集，用于确定MEWMA控制限（CL）。本文保留了与现有方法相同的基于得分的MEWMA监控统计量，重点改进了控制限的计算方法。我们开发了一种新颖的嵌套自举程序来校准控制限，该程序允许使用整个初始样本进行模型拟合，从而在消除对大容量保留集需求的同时，获得更准确的基线模型。我们证明了标准嵌套自举会显著低估监控统计量的变异性，并开发了一种0.632类校正方法以恰当考虑此问题。我们通过数值示例展示了该方法的优势。