Large Language Models(LLMs) have been actively integrated into modern software systems as critical components. LLM-in-the-loop vulnerabilities, where vulnerabilities are introduced by LLMs and their dependent downstream components, such as frameworks, introduce new risks. Although some benchmark datasets have been constructed to study the impact of such vulnerabilities, most works still remain at the analysis from the conventional software level, ignoring the harm actually caused by LLMs. Understanding real-world LLM-in-the-loop vulnerabilities is still an open problem. To address this gap, we build the first LLM-in-the-loop vulnerability dataset, LLMCVE, to facilitate the risk analysis of LLM-integrated software. To do so, we first collect 2,888 multi-source vulnerabilities across 230 popular LLM components. Then, through manual analysis, we identify 205 vulnerabilities that strictly fall under the concept of LLM-in-the-loop vulnerability. Through analysis, we found that LLMs more often play as targets or propagation vectors rather than the root cause of these vulnerabilities. Furthermore, based on LLMCVE, we evaluate the repairing capabilities of existing agent-based vulnerability repair methods, such as SWE-Agent. Experimental results demonstrate that compared to conventional software vulnerabilities, LLM-in-the-Loop vulnerabilities are more challenging to precisely fix, especially for those involving prompt injections where the Pass@1 rate is only 28.57%.

翻译：大语言模型已作为关键组件被积极整合到现代软件系统中。大模型回路漏洞是指由大语言模型及其依赖的下游组件（如框架）引入的新型安全风险。尽管已有基准数据集用于研究此类漏洞的影响，但现有工作大多仍停留在传统软件层面的分析，忽略了由大语言模型实际造成的危害。理解真实世界的大模型回路漏洞仍是一个开放性问题。为填补这一空白，我们构建了首个大模型回路漏洞数据集LLMCVE，以促进大语言模型集成软件的风险分析。为此，我们首先收集了涵盖230个流行大语言模型组件的2888个多源漏洞。通过人工分析，我们筛选出205个严格符合大模型回路漏洞概念的案例。分析发现，大语言模型在这些漏洞中更常扮演目标或传播载体角色，而非根本原因。此外，基于LLMCVE，我们评估了现有基于代理的漏洞修复方法（如SWE-Agent）的修复能力。实验结果表明，与传统软件漏洞相比，大模型回路漏洞更难精准修复，尤其是涉及提示注入的漏洞，其Pass@1率仅为28.57%。