We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.

翻译：我们提出网络防御基准（Cyber Defense Benchmark），用于衡量大型语言模型（LLM）智能体在未经引导性提问或提示的情况下，基于原始Windows事件日志数据库识别恶意事件确切时间戳这一核心安全运营中心（SOC）分析师威胁狩猎任务的表现。该基准将OTRF安全数据集语料库中106个真实攻击流程（涵盖MITRE ATT&CK框架12种战术下的86项子技术）封装为Gymnasium强化学习环境。每个回合中，智能体需面对一个内存型SQLite数据库（含75,000-135,000条日志记录），这些记录由确定性战役模拟器通过时间偏移和实体混淆技术生成。智能体必须通过迭代提交SQL查询来发现恶意事件时间戳并明确标记，最终依据Sigma规则推导的真值进行CTF计分式评估。我们基于26个战役（覆盖106个攻击流程中的105个）对五款前沿模型——Claude Opus 4.6、GPT-5、Gemini 3.1 Pro、Kimi K2.5和Gemini 3 Flash——进行测试，发现所有模型表现均不理想：最佳模型（Claude Opus 4.6）平均仅正确标记3.8%的恶意事件，且没有任何模型在所有运行中完成全部标记。我们将及格线定义为：在每个ATT&CK战术维度上召回率不低于50%（即SOC无监督部署的最低门槛）。结果显示无模型达标：领先模型仅满足13个战术维度中5个的阈值，其余四款模型在所有维度上均未达标。这些结果表明，尽管当前LLM在精心设计的问答式安全基准上表现优异，但面对开放式、证据驱动的威胁狩猎任务时仍存在严重不足。