Penetration testing, a crucial industrial practice for ensuring system security, has traditionally resisted automation due to the extensive expertise required by human professionals. Large Language Models (LLMs) have shown significant advancements in various domains, and their emergent abilities suggest their potential to revolutionize industries. In this research, we evaluate the performance of LLMs on real-world penetration testing tasks using a robust benchmark created from test machines with platforms. Our findings reveal that while LLMs demonstrate proficiency in specific sub-tasks within the penetration testing process, such as using testing tools, interpreting outputs, and proposing subsequent actions, they also encounter difficulties maintaining an integrated understanding of the overall testing scenario. In response to these insights, we introduce PentestGPT, an LLM-empowered automatic penetration testing tool that leverages the abundant domain knowledge inherent in LLMs. PentestGPT is meticulously designed with three self-interacting modules, each addressing individual sub-tasks of penetration testing, to mitigate the challenges related to context loss. Our evaluation shows that PentestGPT not only outperforms LLMs with a task-completion increase of 228.6\% compared to the \gptthree model among the benchmark targets but also proves effective in tackling real-world penetration testing challenges. Having been open-sourced on GitHub, PentestGPT has garnered over 4,700 stars and fostered active community engagement, attesting to its value and impact in both the academic and industrial spheres.

翻译：渗透测试是确保系统安全的关键工业实践，但因需人类专家的专业知识，传统上难以实现自动化。大型语言模型（LLM）在多个领域取得了显著进展，其涌现能力表明它们具备颠覆行业的潜力。本研究利用基于测试平台机器的稳健基准，评估了LLM在真实渗透测试任务中的表现。研究结果表明，尽管LLM在渗透测试的特定子任务（如使用测试工具、解释输出及提出后续操作）中展现出熟练度，但它们在维护整体测试场景的综合理解方面仍面临困难。基于这些发现，我们提出PentestGPT——一种利用LLM丰富领域知识的自动化渗透测试工具。PentestGPT精心设计了三个自交互模块，分别处理渗透测试的各个子任务，以缓解上下文丢失问题。实验评估显示，与基准目标上的GPT-3.5模型相比，PentestGPT的任务完成率提升了228.6%，并且能有效应对真实世界的渗透测试挑战。该工具已在GitHub上开源，获得超过4700个星标并促进了社区积极参与，证明了其在学术和工业领域的价值与影响力。