The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA), a critical part of the incident management process, is a demanding task for on-call engineers, requiring deep domain knowledge and extensive experience with a team's specific services. Automation of RCA can result in significant savings of time, and ease the burden of incident management on on-call engineers. Recently, researchers have utilized Large Language Models (LLMs) to perform RCA, and have demonstrated promising results. However, these approaches are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases, severely restricting their ability to diagnose root causes. In this work, we explore the use of LLM based agents for RCA to address this limitation. We present a thorough empirical evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at Microsoft. Results show that ReAct performs competitively with strong retrieval and reasoning baselines, but with highly increased factual accuracy. We then extend this evaluation by incorporating discussions associated with incident reports as additional inputs for the models, which surprisingly does not yield significant performance improvements. Lastly, we conduct a case study with a team at Microsoft to equip the ReAct agent with tools that give it access to external diagnostic services that are used by the team for manual RCA. Our results show how agents can overcome the limitations of prior work, and practical considerations for implementing such a system in practice.

翻译：云原生软件系统日益增长的复杂性，导致事件管理已成为软件开发周期中不可分割的组成部分。根因分析作为事件管理流程的关键环节，对值班工程师而言是一项极具挑战的任务，需要深厚的领域知识以及对团队特定服务的丰富经验。自动化根因分析可显著节省时间，减轻值班工程师处理事件管理的负担。近期研究者已尝试利用大语言模型执行根因分析，并展示了令人瞩目的成果。然而，现有方法无法动态采集事件相关日志、指标或数据库等额外诊断信息，严重制约了其诊断根因的能力。本研究探索使用基于大语言模型的智能体进行根因分析以解决此局限。我们针对配备检索工具的反应式智能体，在微软收集的生产环境事件分布外数据集上开展了全面的实证评估。结果表明反应式智能体在强检索与推理基线方案中表现优异，同时显著提升了事实准确性。我们进一步将事件报告相关讨论作为模型额外输入进行扩展评估，但令人意外的是，该举措并未带来显著的性能提升。最后，我们与微软某团队开展案例研究，为反应式智能体配备可接入该团队人工根因分析时使用的外部诊断服务工具。研究结果展示了智能体如何克服先前工作的局限，以及在实际系统中实现此类系统的实践考量。