Large Language Models (LLMs) have been used in cybersecurity in many ways, including their recent use as intelligent agent systems for autonomous security analysis. Capture the Flag (CTF) challenges serve as benchmarks for assessing the automated task-planning abilities of LLM agents across various cybersecurity skill sets. Early attempts to apply LLMs for solving CTF challenges relied on single-agent systems, where feedback was restricted to a single reasoning-action loop. This approach proved inadequate for handling complex CTF tasks. Drawing inspiration from real-world CTF competitions, where teams of experts collaborate, we introduce the D-CIPHER multi-agent LLM framework for collaborative CTF challenge solving. D-CIPHER integrates agents with distinct roles, enabling dynamic feedback loops to enhance reasoning on CTF challenges. It introduces the Planner-Executor agent system, consisting of a Planner agent for overall problem-solving along with multiple heterogeneous Executor agents for individual tasks, facilitating efficient allocation of responsibilities among the LLMs. Additionally, D-CIPHER incorporates an Auto-prompter agent, which improves problem-solving by exploring the challenge environment and generating a highly relevant initial prompt. We evaluate D-CIPHER on CTF benchmarks using multiple LLM models and conduct comprehensive studies to highlight the impact of our enhancements. Our results demonstrate that the multi-agent D-CIPHER system achieves a significant improvement in challenges solved, setting a state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox. D-CIPHER is available at https://github.com/NYU-LLM-CTF/nyuctf_agents as the nyuctf_multiagent package.

翻译：大型语言模型（LLM）在网络安全领域已有多种应用，其中近期兴起的智能体系统被用于自主安全分析。夺旗赛（CTF）挑战可作为评估LLM智能体跨多种网络安全技能自动化任务规划能力的基准。早期尝试使用LLM解决CTF挑战主要依赖单智能体系统，其反馈机制局限于单一推理-行动循环，该方法在处理复杂CTF任务时存在明显不足。受现实世界中专家团队协作参与CTF竞赛的启发，我们提出了用于协作式CTF挑战求解的D-CIPHER多智能体LLM框架。D-CIPHER整合了具有不同角色的智能体，通过动态反馈循环增强对CTF挑战的推理能力。该框架引入了规划者-执行者智能体系统，包含一个负责整体问题求解的规划者智能体与多个执行特定任务的异构执行者智能体，实现了LLM间职责的高效分配。此外，D-CIPHER还集成了自动提示生成智能体，通过探索挑战环境并生成高度相关的初始提示来提升问题求解效率。我们在CTF基准测试中使用多种LLM模型对D-CIPHER进行评估，并通过综合研究验证了所提增强策略的有效性。实验结果表明，多智能体D-CIPHER系统在挑战解决率上取得显著提升，在三个基准测试中创造了最先进性能：NYU CTF Bench达到22.0%，Cybench达到22.5%，HackTheBox达到44.0%。D-CIPHER已作为nyuctf_multiagent软件包发布于https://github.com/NYU-LLM-CTF/nyuctf_agents。