Cyber Threat Intelligence (CTI) reports document observations of cyber threats, synthesizing evidence about adversaries' actions and intent into actionable knowledge that informs detection, response, and defense planning. However, the unstructured and verbose nature of CTI reports poses significant challenges for security practitioners to manually extract and analyze such sequences. Although large language models (LLMs) exhibit promise in cybersecurity tasks such as entity extraction and knowledge graph construction, their understanding and reasoning capabilities towards behavioral sequences remains underexplored. To address this, we introduce AttackSeqBench, a benchmark designed to systematically evaluate LLMs' reasoning abilities across the tactical, technical, and procedural dimensions of adversarial behaviors, while satisfying Extensibility, Reasoning Scalability, and Domain-dpecific Epistemic Expandability. We further benchmark 7 LLMs, 5 LRMs and 4 post-training strategies across 3 benchmark settings and 3 benchmark tasks within our AttackSeqBench to identify their advantages and limitations in such specific domain. Our findings contribute to a deeper understanding of LLM-driven CTI report understanding and foster its application in cybersecurity operations. Our code of benchmark construction and evaluation and the corresponding dataset are available at: https://github.com/hulkima/AttackSeqBench.

翻译：网络威胁情报（CTI）报告记录了网络威胁的观测情况，将关于对手行动和意图的证据综合为可操作知识，为检测、响应和防御规划提供信息。然而，CTI报告的非结构化和冗长特性给安全从业者手动提取和分析此类序列带来了重大挑战。尽管大型语言模型（LLMs）在实体提取和知识图谱构建等网络安全任务中展现出潜力，但其对行为序列的理解和推理能力仍未得到充分探索。为此，我们提出了AttackSeqBench，这是一个旨在系统评估LLMs在对抗行为的战术、技术和程序维度上推理能力的基准，同时满足可扩展性、推理可扩展性和领域特定认知可扩展性。我们进一步在AttackSeqBench中，通过3个基准设置和3个基准任务，对7个LLMs、5个LRMs和4种训练后策略进行了基准测试，以识别它们在此特定领域的优势和局限性。我们的研究结果有助于更深入地理解LLM驱动的CTI报告理解，并促进其在网络安全运营中的应用。我们的基准构建与评估代码及相应数据集可在以下网址获取：https://github.com/hulkima/AttackSeqBench。