Security operation centers (SOCs) often produce analysis reports on security incidents, and large language models (LLMs) will likely be used for this task in the near future. We postulate that a better understanding of how veteran analysts evaluate reports, including their feedback, can help produce analysis reports in SOCs. In this paper, we aim to leverage LLMs for analysis reports. To this end, we first construct a Analyst-wise checklist to reflect SOC practitioners' opinions for analysis report evaluation through literature review and user study with SOC practitioners. Next, we design a novel LLM-based conceptual framework, named MESSALA, by further introducing two new techniques, granularization guideline and multi-perspective evaluation. MESSALA can maximize report evaluation and provide feedback on veteran SOC practitioners' perceptions. When we conduct extensive experiments with MESSALA, the evaluation results by MESSALA are the closest to those of veteran SOC practitioners compared with the existing LLM-based methods. We then show two key insights. We also conduct qualitative analysis with MESSALA, and then identify that MESSALA can provide actionable items that are necessary for improving analysis reports.

翻译：安全运营中心（SOC）通常会产生关于安全事件的分析报告，而大型语言模型（LLMs）在不远的将来很可能被用于完成这项任务。我们认为，更深入地理解资深分析师如何评价报告（包括其反馈）有助于在SOC中生成分析报告。本文旨在利用LLMs进行报告分析。为此，我们首先通过文献综述和与SOC从业者的用户研究，构建了一份反映SOC从业人员观点、用于分析报告评估的分析师检查清单。接着，我们设计了一种新颖的基于LLM的概念框架，命名为MESSALA，进一步引入了两项新技术：粒度化指南和多视角评估。MESSALA能够最大化报告评估效果，并提供符合资深SOC从业者认知的反馈。当我们使用MESSALA进行大量实验时，与现有基于LLM的方法相比，MESSALA的评估结果与资深SOC从业者的评估结果最为接近。随后，我们揭示了两项关键见解。我们还使用MESSALA进行了定性分析，并发现MESSALA能够提供改进分析报告所必需的可行建议项。