Penetration testing is critical for identifying and mitigating security vulnerabilities, yet traditional approaches remain expensive, time-consuming, and dependent on expert human labor. Recent work has explored AI-driven pentesting agents, but their evaluation relies on oversimplified capture-the-flag (CTF) settings that embed prior knowledge and reduce complexity, leading to performance estimates far from real-world practice. We close this gap by introducing the first real-world, agent-oriented pentesting benchmark, TermiBench, which shifts the goal from 'flag finding' to achieving full system control. The benchmark spans 510 hosts across 25 services and 30 CVEs, with realistic environments that require autonomous reconnaissance, discrimination between benign and exploitable services, and robust exploit execution. Using this benchmark, we find that existing systems can hardly obtain system shells under realistic conditions. To address these challenges, we propose TermiAgent, a multi-agent penetration testing framework. TermiAgent mitigates long-context forgetting with a Located Memory Activation mechanism and builds a reliable exploit arsenal via structured code understanding rather than naive retrieval. In evaluations, our work outperforms state-of-the-art agents, exhibiting stronger penetration testing capability, reducing execution time and financial cost, and demonstrating practicality even on laptop-scale deployments. Our work delivers both the first open-source benchmark for real-world autonomous pentesting and a novel agent framework that establishes a milestone for AI-driven penetration testing.

翻译：渗透测试对于识别和缓解安全漏洞至关重要，然而传统方法仍然昂贵、耗时且依赖于专家人力。近期研究探索了AI驱动的渗透测试智能体，但其评估依赖于过度简化的夺旗赛（CTT）环境，这些环境嵌入了先验知识并降低了复杂性，导致性能评估与现实实践相去甚远。我们通过引入首个面向现实世界、以智能体为导向的渗透测试基准TermiBench来弥合这一差距，该基准将目标从“寻找旗帜”转变为实现完全系统控制。该基准涵盖25种服务和30个CVE漏洞，涉及510台主机，其真实环境要求自主侦察、区分良性服务与可被利用的服务，以及稳健的漏洞利用执行。使用该基准，我们发现现有系统在现实条件下几乎无法获得系统外壳。为应对这些挑战，我们提出了TermiAgent，一个多智能体渗透测试框架。TermiAgent通过定位记忆激活机制缓解长上下文遗忘问题，并通过结构化代码理解而非简单检索来构建可靠的漏洞利用库。在评估中，我们的工作优于最先进的智能体，展现出更强的渗透测试能力，减少了执行时间和财务成本，并证明了即使在笔记本电脑规模的部署中也具有实用性。我们的工作既提供了首个面向现实世界自主渗透测试的开源基准，也提出了一种新颖的智能体框架，为AI驱动的渗透测试树立了里程碑。