CI/CD pipeline failure management is time-consuming when performed manually. Automating this process is non-trivial because the information required for effective failure management is unstructured and cannot be automatically processed by traditional programs. With their ability to process unstructured data, large language models (LLMs) have shown promising results for automated failure management by previous work. Following these studies, we evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP HANA. We evaluated the ability of the LLM-based system to identify the error location and to propose exact solutions that contain no unnecessary actions. To support the LLM in generating exact solutions, we provided it with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures. We conducted an ablation study to determine which type of domain knowledge contributed most to solution accuracy. The results show that data from historical failures contributed the most to the system's accuracy, enabling it to produce exact solutions in 92.1% of cases in our dataset. The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it. In conclusion, our findings indicate that LLMs, when provided with data from historical failures, represent a promising approach for automating CI/CD pipeline failure management.

翻译：人工执行CI/CD流水线故障管理耗时费力。由于有效故障管理所需的信息是非结构化的，无法被传统程序自动处理，因此实现该过程的自动化并非易事。凭借处理非结构化数据的能力，大型语言模型（LLMs）在先前研究中已展现出自动化故障管理的良好前景。基于这些研究，我们在一个大型工业软件项目（即SAP HANA）的背景下，评估了基于LLM的系统能否自动化CI/CD流水线的故障管理。我们评估了该LLM系统在识别错误位置以及提出不含冗余操作的确切解决方案方面的能力。为支持LLM生成精确解决方案，我们为其提供了多种类型的领域知识，包括流水线信息、故障管理指南以及历史故障数据。我们通过消融实验来确定哪类领域知识对解决方案准确性贡献最大。结果表明，历史故障数据对系统准确性的提升最为显著，使其能在我们数据集中92.1%的案例中生成确切解决方案。在提供领域知识的情况下，系统正确识别错误位置的准确率达到97.4%，而未提供时仅为84.2%。综上所述，我们的研究结果表明，当配备历史故障数据时，LLMs为自动化CI/CD流水线故障管理提供了一种极具前景的途径。