Penetration testing, a crucial industrial practice for ensuring system security, has traditionally resisted automation due to the extensive expertise required by human professionals. Large Language Models (LLMs) have shown significant advancements in various domains, and their emergent abilities suggest their potential to revolutionize industries. In this research, we evaluate the performance of LLMs on real-world penetration testing tasks using a robust benchmark created from test machines with platforms. Our findings reveal that while LLMs demonstrate proficiency in specific sub-tasks within the penetration testing process, such as using testing tools, interpreting outputs, and proposing subsequent actions, they also encounter difficulties maintaining an integrated understanding of the overall testing scenario. In response to these insights, we introduce PentestGPT, an LLM-empowered automatic penetration testing tool that leverages the abundant domain knowledge inherent in LLMs. PentestGPT is meticulously designed with three self-interacting modules, each addressing individual sub-tasks of penetration testing, to mitigate the challenges related to context loss. Our evaluation shows that PentestGPT not only outperforms LLMs with a task-completion increase of 228.6\% compared to the \gptthree model among the benchmark targets but also proves effective in tackling real-world penetration testing challenges. Having been open-sourced on GitHub, PentestGPT has garnered over 4,700 stars and fostered active community engagement, attesting to its value and impact in both the academic and industrial spheres.

翻译：渗透测试作为确保系统安全性的关键工业实践，传统上因高度依赖人类专家的专业知识而难以实现自动化。大语言模型（LLMs）在多个领域已展现出显著进展，其涌现能力表明其具备革新行业的潜力。本研究基于从测试平台构建的稳健基准，评估了LLMs在真实世界渗透测试任务中的表现。研究发现，尽管LLMs在渗透测试流程的特定子任务（如使用测试工具、解析输出结果及提出后续行动建议）中表现出熟练能力，但在维持对整体测试场景的综合理解方面仍存在困难。基于这些发现，我们提出了PentestGPT——一种利用LLMs丰富领域知识的自动化渗透测试工具。PentestGPT精心设计了三个自交互模块，每个模块分别处理渗透测试的独立子任务，以缓解上下文丢失带来的挑战。评估结果表明，PentestGPT不仅在基准测试目标中优于LLMs（相较于\gptthree模型，任务完成率提升228.6%），还能有效应对真实世界的渗透测试挑战。该工具已在GitHub开源，获得超过4,700个星标并促进了活跃的社区参与，印证了其在学术与工业领域的价值与影响力。