Large language models (LLMs) are increasingly deployed as the service backend for LLM-integrated applications such as code completion and AI-powered search. LLM-integrated applications serve as middleware to refine users' queries with domain-specific knowledge to better inform LLMs and enhance the responses. Despite numerous opportunities and benefits, LLM-integrated applications also introduce new attack surfaces. Understanding, minimizing, and eliminating these emerging attack surfaces is a new area of research. In this work, we consider a setup where the user and LLM interact via an LLM-integrated application in the middle. We focus on the communication rounds that begin with user's queries and end with LLM-integrated application returning responses to the queries, powered by LLMs at the service backend. For this query-response protocol, we identify potential vulnerabilities that can originate from the malicious application developer or from an outsider threat initiator that is able to control the database access, manipulate and poison data that are high-risk for the user. Successful exploits of the identified vulnerabilities result in the users receiving responses tailored to the intent of a threat initiator. We assess such threats against LLM-integrated applications empowered by OpenAI GPT-3.5 and GPT-4. Our empirical results show that the threats can effectively bypass the restrictions and moderation policies of OpenAI, resulting in users receiving responses that contain bias, toxic content, privacy risk, and disinformation. To mitigate those threats, we identify and define four key properties, namely integrity, source identification, attack detectability, and utility preservation, that need to be satisfied by a safe LLM-integrated application. Based on these properties, we develop a lightweight, threat-agnostic defense that mitigates both insider and outsider threats.

翻译：大型语言模型（LLM）日益被部署为代码补全、AI增强搜索等LLM集成应用的后端服务。LLM集成应用作为中间件，通过注入领域特定知识来优化用户查询，从而提升LLM的理解能力与响应质量。尽管带来了众多机遇与益处，这类应用也催生了新的攻击面。理解、最小化并消除这些新兴攻击面已成为新兴研究领域。本研究考虑用户与LLM通过中间应用进行交互的配置场景，重点关注从用户查询发起至LLM集成应用在服务后端依靠LLM返回结果的通信周期。针对该查询-响应协议，我们识别出两类潜在漏洞：恶意应用开发者引发的内部威胁，以及能控制数据库访问、篡改或污染用户高风险数据的外部威胁发起者。成功利用这些漏洞将导致用户接收到符合威胁发起者意图的响应。我们通过OpenAI GPT-3.5和GPT-4驱动的LLM集成应用评估此类威胁，实验表明威胁可有效绕过OpenAI的约束与内容审核策略，使用户接收到包含偏见、有害内容、隐私风险及虚假信息的响应。为缓解威胁，我们识别并定义了安全LLM集成应用需满足的四个关键属性：完整性、源标识、攻击可检测性与效用保持。基于这些属性，我们开发了轻量级且不依赖特定威胁的防御机制，可同时缓解内部与外部威胁。