Automated log analysis is crucial in modern software-intensive systems for facilitating program comprehension throughout software maintenance and engineering life cycles. Existing methods perform tasks such as log parsing and log anomaly detection by providing a single prediction value without interpretation. However, given the increasing volume of system events, the limited interpretability of analysis results hinders analysts' comprehension of program status and their ability to take appropriate actions. Moreover, these methods require substantial in-domain training data, and their performance declines sharply (by up to 62.5%) in online scenarios involving unseen logs from new domains, a common occurrence due to rapid software updates. In this paper, we propose LogPrompt, a novel interpretable log analysis approach for online scenarios. LogPrompt employs large language models (LLMs) to perform online log analysis tasks via a suite of advanced prompt strategies tailored for log tasks, which enhances LLMs' performance by up to 380.7% compared with simple prompts. Experiments on nine publicly available evaluation datasets across two tasks demonstrate that LogPrompt, despite requiring no in-domain training, outperforms existing approaches trained on thousands of logs by up to 55.9%. We also conduct a human evaluation of LogPrompt's interpretability, with six practitioners possessing over 10 years of experience, who highly rated the generated content in terms of usefulness and readability (averagely 4.42/5). LogPrompt also exhibits remarkable compatibility with open-source and smaller-scale LLMs, making it flexible for practical deployment. Code of LogPrompt is available at https://github.com/lunyiliu/LogPrompt.
翻译:自动日志分析对于现代软件密集型系统至关重要,有助于在软件维护和工程生命周期中促进程序理解。现有方法通过提供单一预测值(无解释)来执行日志解析和日志异常检测等任务。然而,随着系统事件数量的不断增加,分析结果有限的解释性阻碍了分析人员对程序状态的理解以及采取适当行动的能力。此外,这些方法需要大量的领域内训练数据,并且在涉及来自新领域的未见日志的在线场景中(由于软件快速更新,这种情况很常见),其性能会急剧下降(高达62.5%)。本文提出了一种面向在线场景的新型可解释日志分析方法LogPrompt。LogPrompt利用大语言模型(LLMs),通过一套针对日志任务定制的先进提示策略来执行在线日志分析任务,与简单提示相比,可将LLMs的性能提升高达380.7%。在两个任务的九个公开评估数据集上的实验表明,LogPrompt尽管不需要领域内训练,但在经过数千条日志训练的现有方法上性能仍可提升高达55.9%。我们还对LogPrompt的可解释性进行了人工评估,六位拥有超过10年经验的从业者对生成内容的有用性和可读性给予了高度评价(平均4.42/5分)。LogPrompt还表现出与开源及小规模LLMs的出色兼容性,使其在实际部署中具有灵活性。LogPrompt的代码可在https://github.com/lunyiliu/LogPrompt获取。