Anomaly detection in distributed systems such as High-Performance Computing (HPC) clusters is vital for early fault detection, performance optimisation, security monitoring, reliability in general but also operational insights. Deep Neural Networks have seen successful use in detecting long-term anomalies in multidimensional data, originating for instance from industrial or medical systems, or weather prediction. A downside of such methods is that they require a static input size, or lose data through cropping, sampling, or other dimensionality reduction methods, making deployment on systems with variability on monitored data channels, such as computing clusters difficult. To address these problems, we present DeepHYDRA (Deep Hybrid DBSCAN/Reduction-Based Anomaly Detection) which combines DBSCAN and learning-based anomaly detection. DBSCAN clustering is used to find point anomalies in time-series data, mitigating the risk of missing outliers through loss of information when reducing input data to a fixed number of channels. A deep learning-based time-series anomaly detection method is then applied to the reduced data in order to identify long-term outliers. This hybrid approach reduces the chances of missing anomalies that might be made indistinguishable from normal data by the reduction process, and likewise enables the algorithm to be scalable and tolerate partial system failures while retaining its detection capabilities. Using a subset of the well-known SMD dataset family, a modified variant of the Eclipse dataset, as well as an in-house dataset with a large variability in active data channels, made publicly available with this work, we furthermore analyse computational intensity, memory footprint, and activation counts. DeepHYDRA is shown to reliably detect different types of anomalies in both large and complex datasets.

翻译：分布式系统（如高性能计算集群）中的异常检测对于早期故障发现、性能优化、安全监控、整体可靠性保障以及运维洞察至关重要。深度神经网络已成功应用于多维度数据（例如工业、医疗系统或天气预报数据）中的长期异常检测。然而此类方法的局限性在于：需要固定尺寸的静态输入，否则需通过裁剪、采样或其他降维方法导致数据损失，这使得其难以部署在监控数据通道存在动态变化的系统（如计算集群）中。为解决上述问题，我们提出DeepHYDRA（深度混合DBSCAN/基于约简的异常检测），该方法融合DBSCAN与基于学习的异常检测技术。首先利用DBSCAN聚类识别时间序列数据中的点异常，从而规避因将输入数据降维至固定通道数时信息丢失而遗漏异常值的风险；随后对降维后的数据应用基于深度学习的时间序列异常检测方法，以识别长期异常值。这种混合策略既降低了因降维过程将异常特征混淆为正常数据而漏检的风险，又使算法具备可扩展性，能在保持检测能力的同时容忍部分系统故障。基于公开的SMD数据集子集、Eclipse数据集的改良变体，以及本研究公开的自有高通道动态变化数据集（该数据集具有活跃数据通道的大幅波动特征），我们从计算强度、内存占用及激活次数三个维度进行了分析。实验表明，DeepHYDRA能够可靠地检测大规模复杂数据集中的多种异常类型。