Most enterprise applications use logging as a mechanism to diagnose anomalies, which could help with reducing system downtime. Anomaly detection using software execution logs has been explored in several prior studies, using both classical and deep neural network-based machine learning models. In recent years, the research has largely focused in using variations of sequence-based deep neural networks (e.g., Long-Short Term Memory and Transformer-based models) for log-based anomaly detection on open-source data. However, they have not been applied in industrial datasets, as often. In addition, the studied open-source datasets are typically very large in size with logging statements that do not change much over time, which may not be the case with a dataset from an industrial service that is relatively new. In this paper, we evaluate several state-of-the-art anomaly detection models on an industrial dataset from our research partner, which is much smaller and loosely structured than most large scale open-source benchmark datasets. Results show that while all models are capable of detecting anomalies, certain models are better suited for less-structured datasets. We also see that model effectiveness changes when a common data leak associated with a random train-test split in some prior work is removed. A qualitative study of the defects' characteristics identified by the developers on the industrial dataset further shows strengths and weaknesses of the models in detecting different types of anomalies. Finally, we explore the effect of limited training data by gradually increasing the training set size, to evaluate if the model effectiveness does depend on the training set size.

翻译：大多数企业应用程序使用日志记录作为诊断异常的机制，这有助于减少系统停机时间。利用软件执行日志进行异常检测已在多项先前研究中得到探索，这些研究采用了基于经典机器学习和深度神经网络的模型。近年来，研究主要集中在使用基于序列的深度神经网络变体（例如长短期记忆网络和基于Transformer的模型）对开源数据进行日志异常检测。然而，这些方法尚未广泛应用于工业数据集。此外，所研究的开源数据集通常规模庞大，且日志语句随时间变化不大，这与相对较新的工业服务数据集可能有所不同。在本文中，我们评估了多个最先进的异常检测模型，这些模型应用于研究合作伙伴提供的工业数据集，该数据集比大多数大规模开源基准数据集更小且结构更松散。结果表明，尽管所有模型都能检测异常，但某些模型更适合处理结构松散的数据集。我们还发现，当移除先前某些工作中因随机划分训练集和测试集而导致的常见数据泄露时，模型的有效性会发生变化。对工业数据集中开发者识别的缺陷特征进行的定性研究进一步揭示了模型在检测不同类型异常时的优势与不足。最后，我们通过逐步增加训练集规模来探索有限训练数据的影响，以评估模型有效性是否确实依赖于训练集大小。