Real-world offensive security operations are inherently open-ended: attackers explore unknown attack surfaces, revise hypotheses under uncertainty, and operate without guaranteed success. Existing LLM-based offensive agent evaluations rely on closed-world settings with predefined goals and binary success criteria. To address this gap, we introduce CyberExplorer, an evaluation suite with two core components: (1) an open-environment benchmark built on a virtual machine hosting 40 vulnerable web services derived from real-world CTF challenges, where agents autonomously perform reconnaissance, target selection, and exploitation without prior knowledge of vulnerability locations; and (2) a reactive multi-agent framework supporting dynamic exploration without predefined plans. CyberExplorer enables fine-grained evaluation beyond flag recovery, capturing interaction dynamics, coordination behavior, failure modes, and vulnerability discovery signals-bridging the gap between benchmarks and realistic multi-target attack scenarios.

翻译：现实世界的攻击性安全操作本质上是开放式的：攻击者探索未知的攻击面，在不确定性下修正假设，并且在没有成功保证的情况下展开行动。现有基于大语言模型的攻击代理评估依赖于封闭世界设定，具有预定义的目标和二元化的成功标准。为弥补这一差距，我们引入了CyberExplorer，这是一个包含两个核心组件的评估套件：(1) 一个基于虚拟机构建的开放环境基准测试，该虚拟机托管了40个源自现实世界CTF挑战的易受攻击的Web服务，代理在不知晓漏洞位置先验知识的情况下，自主执行侦察、目标选择和漏洞利用；(2) 一个支持动态探索、无需预定义计划的反应式多代理框架。CyberExplorer实现了超越标志（flag）获取的细粒度评估，能够捕捉交互动态、协调行为、失败模式以及漏洞发现信号，从而弥合了基准测试与真实多目标攻击场景之间的差距。