Automated log analysis is crucial in modern software-intensive systems for ensuring reliability and resilience throughout software maintenance and engineering life cycles. Existing methods perform tasks such as log parsing and log anomaly detection by providing a single prediction value without interpretation. However, given the increasing volume of system events, the limited interpretability of analysis results hinders analysts' trust and their ability to take appropriate actions. Moreover, these methods require substantial in-domain training data, and their performance declines sharply (by up to 62.5%) in online scenarios involving unseen logs from new domains, a common occurrence due to rapid software updates. In this paper, we propose LogPrompt, a novel zero-shot and interpretable log analysis approach. LogPrompt employs large language models (LLMs) to perform zero-shot log analysis tasks via a suite of advanced prompt strategies tailored for log tasks, which enhances LLMs' performance by up to 107.5% compared with simple prompts. Experiments on nine publicly available evaluation datasets across two tasks demonstrate that LogPrompt, despite using no training data, outperforms existing approaches trained on thousands of logs by up to around 50%. We also conduct a human evaluation of LogPrompt's interpretability, with six practitioners possessing over 10 years of experience, who highly rated the generated content in terms of usefulness and readability (averagely 4.42/5). LogPrompt also exhibits remarkable compatibility with open-source and smaller-scale LLMs, making it flexible for practical deployment.
翻译:自动化日志分析对于现代软件密集型系统在软件维护与工程全生命周期中确保可靠性和弹性至关重要。现有方法通过提供单一预测值(无解释)来执行日志解析和日志异常检测等任务。然而,随着系统事件数量不断增加,分析结果的可解释性有限,这阻碍了分析人员的信任及其采取适当行动的能力。此外,这些方法需要大量领域内训练数据,且在新领域未见日志的在线场景中(因软件快速更新而常见),其性能急剧下降(最高达62.5%)。本文提出LogPrompt——一种新颖的零样本、可解释日志分析方法。LogPrompt利用大语言模型,通过一套专为日志任务定制的高级提示策略来执行零样本日志分析任务,与简单提示相比,该策略能将大语言模型的性能提升高达107.5%。在两个任务的九个公开评估数据集上的实验表明,LogPrompt尽管不使用任何训练数据,其性能仍比基于数千条日志训练的现有方法高出约50%。我们还对LogPrompt的可解释性进行了人工评估,六位拥有超过10年经验的从业者对生成内容的有用性和可读性给予了高度评价(平均4.42/5分)。LogPrompt还表现出与开源及较小规模大语言模型的出色兼容性,使其在实际部署中具有灵活性。