Large Language Models (LLMs) such as ChatGPT and Llama-2 have become prevalent in real-world applications, exhibiting impressive text generation performance. LLMs are fundamentally developed from a scenario where the input data remains static and lacks a clear structure. To behave interactively over time, LLM-based chat systems must integrate additional contextual information (i.e., chat history) into their inputs, following a pre-defined structure. This paper identifies how such integration can expose LLMs to misleading context from untrusted sources and fail to differentiate between system and user inputs, allowing users to inject context. We present a systematic methodology for conducting context injection attacks aimed at eliciting disallowed responses by introducing fabricated context. This could lead to illegal actions, inappropriate content, or technology misuse. Our context fabrication strategies, acceptance elicitation and word anonymization, effectively create misleading contexts that can be structured with attacker-customized prompt templates, achieving injection through malicious user messages. Comprehensive evaluations on real-world LLMs such as ChatGPT and Llama-2 confirm the efficacy of the proposed attack with success rates reaching 97%. We also discuss potential countermeasures that can be adopted for attack detection and developing more secure models. Our findings provide insights into the challenges associated with the real-world deployment of LLMs for interactive and structured data scenarios.

翻译：大型语言模型（LLMs）如ChatGPT和Llama-2已在现实应用中广泛部署，展现出卓越的文本生成性能。LLMs本质上是在输入数据保持静态且缺乏明确结构的场景下开发的。为实现随时间推移的交互行为，基于LLM的聊天系统必须按照预定义结构将额外的上下文信息（即聊天历史）整合到输入中。本文揭示了此类整合如何使LLMs面临来自不可信源的误导性上下文，且无法区分系统输入与用户输入，从而允许用户注入上下文。我们提出了一种系统化的上下文注入攻击方法，旨在通过引入伪造上下文来诱导模型生成被禁止的响应，可能导致非法行为、不当内容或技术滥用。我们提出的上下文伪造策略——接受诱导与词汇匿名化——能有效构建误导性上下文，这些上下文可通过攻击者定制的提示模板进行结构化，并通过恶意用户消息实现注入。在ChatGPT和Llama-2等现实世界LLMs上的综合评估证实了所提攻击的有效性，成功率高达97%。我们还探讨了可用于攻击检测和开发更安全模型的潜在防御措施。本研究为LLMs在交互式结构化数据场景中的实际部署所面临的挑战提供了重要见解。