Penetration testing, an essential component of cybersecurity, allows organizations to proactively identify and remediate vulnerabilities in their systems, thus bolstering their defense mechanisms against potential cyberattacks. One recent advancement in the realm of penetration testing is the utilization of Language Models (LLMs). We explore the intersection of LLMs and penetration testing to gain insight into their capabilities and challenges in the context of privilige escalation. We create an automated Linux privilege-escalation benchmark utilizing local virtual machines. We introduce an LLM-guided privilege-escalation tool designed for evaluating different LLMs and prompt strategies against our benchmark. We analyze the impact of different prompt designs, the benefits of in-context learning, and the advantages of offering high-level guidance to LLMs. We discuss challenging areas for LLMs, including maintaining focus during testing, coping with errors, and finally comparing them with both stochastic parrots as well as with human hackers.

翻译：渗透测试作为网络安全的关键组成部分，使组织能够主动识别和修复其系统中的漏洞，从而增强对潜在网络攻击的防御机制。在渗透测试领域的最新进展之一是利用语言模型（LLMs）。我们探究了LLMs与渗透测试的交汇点，以深入了解它们在权限提升背景下的能力与挑战。我们利用本地虚拟机创建了一个自动化Linux权限提升基准测试工具。我们引入了一种基于LLM的权限提升工具，旨在评估不同LLMs和提示策略在该基准测试中的表现。我们分析了不同提示设计的影响、上下文学习的益处，以及向LLMs提供高级指导的优势。我们讨论了LLMs面临的挑战领域，包括在测试过程中保持专注、应对错误，并最终将其与随机鹦鹉及人类黑客进行比较。