For many AI systems, concept drift detection is crucial to ensure the systems reliability. These systems often have to deal with large amounts of data or react in real time. Thus, drift detectors must meet computational requirements or constraints with a comprehensive performance evaluation. However, so far, the focus of developing drift detectors is on detection quality, e.g.~accuracy, but not on computational performance, such as running time. We show that the previous works consider computational performance only as a secondary objective and do not have a benchmark for such evaluation. Hence, we propose a novel benchmark suite for drift detectors that accounts both detection quality and computational performance to ensure a detector's applicability in various AI systems. In this work, we focus on unsupervised drift detectors that are not restricted to the availability of labeled data and thus being widely applicable. Our benchmark suite supports configurable synthetic and real world data streams. Moreover, it provides means for simulating a machine learning model's output to unify the performance evaluation across different drift detectors. This allows a fair and comprehensive comparison of drift detectors proposed in related work. Our benchmark suite is integrated in the existing framework, Massive Online Analysis (MOA). To evaluate our benchmark suite's capability, we integrate two representative unsupervised drift detectors. Our work enables the scientific community to achieve a baseline for unsupervised drift detectors with respect to both detection quality and computational performance.

翻译：对于许多人工智能系统而言，概念漂移检测对于确保系统可靠性至关重要。这些系统通常需要处理大量数据或实时响应。因此，漂移检测器必须满足计算要求或约束条件，并进行全面的性能评估。然而，迄今为止，漂移检测器的开发重点在于检测质量（例如准确率），而非计算性能（例如运行时间）。我们表明，先前的研究仅将计算性能视为次要目标，且缺乏此类评估的基准测试。为此，我们提出一种新颖的漂移检测器基准测试套件，该套件同时考虑检测质量与计算性能，以确保检测器在各种人工智能系统中的适用性。本研究聚焦于无监督漂移检测器，其不受限于标注数据的可用性，因而具有广泛适用性。我们的基准测试套件支持可配置的合成数据流与真实世界数据流，并提供模拟机器学习模型输出的机制，以统一不同漂移检测器的性能评估。这使得能够对相关研究中提出的漂移检测器进行公平且全面的比较。该基准测试套件已集成至现有框架Massive Online Analysis（MOA）中。为评估套件能力，我们集成了两种具有代表性的无监督漂移检测器。本研究使科学界能够建立无监督漂移检测器在检测质量与计算性能两方面的基线标准。