Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

翻译：关键性能指标（KPI）是保障众多软件系统可靠性与稳定性的核心时间序列度量。它们如实记录系统运行状态，有助于理解系统异常行为，并为工程师定位根本原因提供有价值的线索。然而，现代软件系统的规模和复杂度前所未有，导致KPI数据量激增。因此，许多传统的KPI异常检测方法变得不切实际，这成为学术界和工业界基于机器学习的解决方案快速发展的催化剂。但目前这些KPI异常检测方法之间缺乏严格的对比，且重新实现需要大量精力。此外，我们注意到不同研究采用独立的评估流程和不同的评估指标。其中一些指标可能无法全面揭示模型能力，另一些则可能造成进展的假象。为更好地理解不同KPI异常检测器的特性并解决评估问题，本文对十二种最先进的方法进行了全面回顾与评估，并提出了一种名为salience的新指标。具体而言，所选方法包括五种基于传统机器学习的方法和七种基于深度学习方法。这些方法在五个公开可用的多元KPI数据集上进行了评估。我们还发布了一个具有易用接口的统一工具包。我们从准确率、显著度、效率与延迟四个维度报告了基准测试结果，这些结果对工业部署具有实际意义。我们相信，本工作可为未来的学术研究与工业应用奠定基础。