Several techniques for multivariate time series anomaly detection have been proposed recently, but a systematic comparison on a common set of datasets and metrics is lacking. This paper presents a systematic and comprehensive evaluation of unsupervised and semi-supervised deep-learning based methods for anomaly detection and diagnosis on multivariate time series data from cyberphysical systems. Unlike previous works, we vary the model and post-processing of model errors, i.e. the scoring functions independently of each other, through a grid of 10 models and 4 scoring functions, comparing these variants to state of the art methods. In time-series anomaly detection, detecting anomalous events is more important than detecting individual anomalous time-points. Through experiments, we find that the existing evaluation metrics either do not take events into account, or cannot distinguish between a good detector and trivial detectors, such as a random or an all-positive detector. We propose a new metric to overcome these drawbacks, namely, the composite F-score ($Fc_1$), for evaluating time-series anomaly detection. Our study highlights that dynamic scoring functions work much better than static ones for multivariate time series anomaly detection, and the choice of scoring functions often matters more than the choice of the underlying model. We also find that a simple, channel-wise model - the Univariate Fully-Connected Auto-Encoder, with the dynamic Gaussian scoring function emerges as a winning candidate for both anomaly detection and diagnosis, beating state of the art algorithms.

翻译：最近提出了数种多变时间序列异常现象探测技术,但缺乏对一套共同的数据集和计量标准进行系统化比较。本文件对未经监督和半监督的深学习方法进行了系统和全面的评价,以从网络物理系统中对多变时间序列数据进行异常检测和诊断。与以往的工作不同,我们通过10个模型和4个评分函数的网格,将这些变量与最新方法的状态进行比较,对模型错误的模型和后处理功能进行了不同。在时间序列异常探测中,发现异常事件比发现个别异常的算法时间点更为重要。通过实验,我们发现现有的评估指标要么没有将事件考虑在内,要么无法区分良好的检测器和微小探测器,例如随机或完全积极的检测器。我们提出了一个新的衡量标准,以克服这些退缩,即复合F-核心(Fc_1美元),用以评价时间序列异常检测。我们的研究强调,动态的评分功能比动态的确定单个反常值的直径比常变的直径(Oral-ral-ral-reval-ral-ral-ral-ral-ral-ral-ration-reval-reval-ration-ration-ral-revation-ration-ral-revation-ral-ral-de-ration-ration-rational-de-ration-rvial-de-de-de-ration-ration-roisal-ration-ration-ration-rence-rmmmmmmmmmmation-rence-smmmmmissation-smmation)的测算、我们找到-smmal-s-s-s-s-s-s-s-s-s-smmal-s-smal-s-s-s-s-s-rmmmmmal-smmmmmmmmal-smmmmmmmmmal-ral-s-s-s-rmmmmmmmmmmmmmmmmmmmmmal-s-rmmmmmmmmmmmmmmmmmmmmmmmmm