Penetration-testing is crucial for identifying system vulnerabilities, with privilege-escalation being a critical subtask to gain elevated access to protected resources. Language Models (LLMs) presents new avenues for automating these security practices by emulating human behavior. However, a comprehensive understanding of LLMs' efficacy and limitations in performing autonomous Linux privilege-escalation attacks remains under-explored. To address this gap, we introduce hackingBuddyGPT, a fully automated LLM-driven prototype designed for autonomous Linux privilege-escalation. We curated a novel, publicly available Linux privilege-escalation benchmark, enabling controlled and reproducible evaluation. Our empirical analysis assesses the quantitative success rates and qualitative operational behaviors of various LLMs -- GPT-3.5-Turbo, GPT-4-Turbo, and Llama3 -- against baselines of human professional pen-testers and traditional automated tools. We investigate the impact of context management strategies, different context sizes, and various high-level guidance mechanisms on LLM performance. Results show that GPT-4-Turbo demonstrates high efficacy, successfully exploiting 33-83% of vulnerabilities, a performance comparable to human pen-testers (75%). In contrast, local models like Llama3 exhibited limited success (0-33%), and GPT-3.5-Turbo achieved moderate rates (16-50%). We show that both high-level guidance and state-management through LLM-driven reflection significantly boost LLM success rates. Qualitative analysis reveals both LLMs' strengths and weaknesses in generating valid commands and highlights challenges in common-sense reasoning, error handling, and multi-step exploitation, particularly with temporal dependencies. Cost analysis indicates that GPT-4-Turbo can achieve human-comparable performance at competitive costs, especially with optimized context management.

翻译：渗透测试对于识别系统漏洞至关重要，其中权限提升是获取受保护资源更高访问权限的关键子任务。语言模型（LLMs）通过模拟人类行为，为自动化这些安全实践提供了新途径。然而，对于LLMs在执行自主Linux权限提升攻击方面的效能与局限性的全面理解仍待深入探索。为填补这一空白，我们提出了hackingBuddyGPT，一个专为自主Linux权限提升设计的全自动LLM驱动原型系统。我们构建了一个新颖、公开可用的Linux权限提升基准测试集，支持受控且可复现的评估。我们的实证分析评估了多种LLM（GPT-3.5-Turbo、GPT-4-Turbo和Llama3）在定量成功率与定性操作行为上的表现，并与人类专业渗透测试员和传统自动化工具基线进行了对比。我们研究了上下文管理策略、不同上下文规模及多种高层指导机制对LLM性能的影响。结果表明，GPT-4-Turbo展现出高效能，成功利用了33-83%的漏洞，其表现与人类渗透测试员（75%）相当。相比之下，Llama3等本地模型成功率有限（0-33%），GPT-3.5-Turbo取得了中等成功率（16-50%）。我们证明，通过LLM驱动反思实现的高层指导与状态管理能显著提升LLM的成功率。定性分析揭示了LLM在生成有效命令方面的优势与不足，并凸显了其在常识推理、错误处理及多步骤利用（尤其涉及时间依赖关系时）方面面临的挑战。成本分析表明，GPT-4-Turbo能以具有竞争力的成本实现与人类相当的性能，特别是在优化上下文管理的情况下。