Real-world offensive security operations are inherently open-ended: attackers explore unknown attack surfaces, revise hypotheses under uncertainty, and operate without guaranteed success. Existing LLM-based offensive agent evaluations rely on closed-world settings with predefined goals and binary success criteria. To address this gap, we introduce CyberExplorer, an evaluation suite with two core components: (1) an open-environment benchmark built on a virtual machine hosting 40 vulnerable web services derived from real-world CTF challenges, where agents autonomously perform reconnaissance, target selection, and exploitation without prior knowledge of vulnerability locations; and (2) a reactive multi-agent framework supporting dynamic exploration without predefined plans. CyberExplorer enables fine-grained evaluation beyond flag recovery, capturing interaction dynamics, coordination behavior, failure modes, and vulnerability discovery signals-bridging the gap between benchmarks and realistic multi-target attack scenarios.

翻译：现实世界的进攻性安全操作本质上是开放式的：攻击者探索未知的攻击面，在不确定性下修正假设，并且在没有成功保证的情况下展开行动。现有基于大型语言模型的进攻性智能体评估依赖于封闭世界设定，其具有预定义的目标和二元化的成功标准。为弥补这一差距，我们提出了CyberExplorer，这是一个包含两个核心组件的评估套件：（1）一个基于虚拟机构建的开放环境基准测试，该虚拟机托管了40个源自现实世界夺旗挑战赛的易受攻击Web服务，智能体可在不知晓漏洞位置先验知识的情况下，自主执行侦察、目标选择和漏洞利用；（2）一个支持动态探索而无需预定义计划的反应式多智能体框架。CyberExplorer实现了超越旗帜获取的细粒度评估，能够捕捉交互动态、协同行为、失败模式以及漏洞发现信号，从而弥合了基准测试与真实多目标攻击场景之间的差距。