Multivariate Time Series (MVTS) anomaly detection is a long-standing and challenging research topic that has attracted tremendous research effort from both industry and academia recently. However, a careful study of the literature makes us realize that 1) the community is active but not as organized as other sibling machine learning communities such as Computer Vision (CV) and Natural Language Processing (NLP), and 2) most proposed solutions are evaluated using either inappropriate or highly flawed protocols, with an apparent lack of scientific foundation. So flawed is one very popular protocol, the so-called point-adjust protocol, that a random guess can be shown to systematically outperform all algorithms developed so far. In this paper, we review and evaluate many recent algorithms using more robust protocols and discuss how a normally good protocol may have weaknesses in the context of MVTS anomaly detection and how to mitigate them. We also share our concerns about benchmark datasets, experiment design and evaluation methodology we observe in many works. Furthermore, we propose a simple, yet challenging, baseline based on Principal Components Analysis (PCA) that surprisingly outperforms many recent Deep Learning (DL) based approaches on popular benchmark datasets. The main objective of this work is to stimulate more effort towards important aspects of the research such as data, experiment design, evaluation methodology and result interpretability, instead of putting the highest weight on the design of increasingly more complex and "fancier" algorithms.

翻译：多元时间序列（MVTS）异常检测是一个长期存在且富有挑战性的研究课题，近年来吸引了工业界和学术界的广泛关注。然而，对文献的仔细研究使我们意识到：1）该研究社区虽活跃，但不如计算机视觉（CV）和自然语言处理（NLP）等姊妹机器学习社区组织有序；2）大多数提出的解决方案采用不恰当或存在严重缺陷的评估协议进行评估，明显缺乏科学基础。一种非常流行的所谓"点调整"（point-adjust）协议甚至存在严重缺陷——随机猜测可以系统性地超越目前所有已开发的算法。在本文中，我们使用更稳健的协议回顾并评估了许多近期算法，探讨了本应良好的协议在多元时间序列异常检测背景下可能存在的弱点及缓解方法。我们还对许多工作中观察到的基准数据集、实验设计和评估方法表示担忧。此外，我们提出一种基于主成分分析（PCA）的简单但具有挑战性的基线方法，令人惊讶的是，该方法在流行基准数据集上超越了众多近期基于深度学习（DL）的方法。本研究的主要目标是推动学界更多关注数据、实验设计、评估方法和结果可解释性等重要研究方面，而非将最高权重放在设计越来越复杂和"花哨"的算法上。