Tool-calling has changed Large Language Model (LLM) applications by integrating external tools, significantly enhancing their functionality across diverse tasks. However, this integration also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. To fill this gap, we present ToolCommander, a novel framework designed to exploit vulnerabilities in LLM tool-calling systems through adversarial tool injection. Our framework employs a well-designed two-stage attack strategy. Firstly, it injects malicious tools to collect user queries, then dynamically updates the injected tools based on the stolen information to enhance subsequent attacks. These stages enable ToolCommander to execute privacy theft, launch denial-of-service attacks, and even manipulate business competition by triggering unscheduled tool-calling. Notably, the ASR reaches 91.67% for privacy theft and hits 100% for denial-of-service and unscheduled tool calling in certain cases. Our work demonstrates that these vulnerabilities can lead to severe consequences beyond simple misuse of tool-calling systems, underscoring the urgent need for robust defensive strategies to secure LLM Tool-calling systems.

翻译：工具调用通过集成外部工具改变了大型语言模型（LLM）的应用生态，显著增强了其在多样化任务中的功能。然而，这种集成也引入了新的安全漏洞，尤其是在LLM的工具调度机制中，这一问题尚未得到充分研究。为填补这一空白，我们提出了ToolCommander——一个通过对抗性工具注入来利用LLM工具调用系统漏洞的新型框架。该框架采用精心设计的双阶段攻击策略：首先注入恶意工具以收集用户查询，随后基于窃取的信息动态更新注入工具以增强后续攻击。这两个阶段使ToolCommander能够实施隐私窃取、发起拒绝服务攻击，甚至通过触发非计划性工具调用来操纵商业竞争。值得注意的是，在某些场景下，隐私窃取攻击的成功率（ASR）达到91.67%，而拒绝服务与非计划性工具调用的攻击成功率可达100%。本研究证明这些漏洞可能导致远超工具调用系统简单误用的严重后果，凸显了构建鲁棒防御策略以保障LLM工具调用系统安全的紧迫性。