Recent remarkable advancements in large language models (LLMs) have led to their widespread adoption in various applications. A key feature of these applications is the combination of LLMs with external content, where user instructions and third-party content are combined to create prompts for LLM processing. These applications, however, are vulnerable to indirect prompt injection attacks, where malicious instructions embedded within external content compromise LLM's output, causing their responses to deviate from user expectations. Despite the discovery of this security issue, no comprehensive analysis of indirect prompt injection attacks on different LLMs is available due to the lack of a benchmark. Furthermore, no effective defense has been proposed. In this work, we introduce the first benchmark, BIPIA, to measure the robustness of various LLMs and defenses against indirect prompt injection attacks. Our experiments reveal that LLMs with greater capabilities exhibit more vulnerable to indirect prompt injection attacks for text tasks, resulting in a higher ASR. We hypothesize that indirect prompt injection attacks are mainly due to the LLMs' inability to distinguish between instructions and external content. Based on this conjecture, we propose four black-box methods based on prompt learning and a white-box defense methods based on fine-tuning with adversarial training to enable LLMs to distinguish between instructions and external content and ignore instructions in the external content. Our experimental results show that our black-box defense methods can effectively reduce ASR but cannot completely thwart indirect prompt injection attacks, while our white-box defense method can reduce ASR to nearly zero with little adverse impact on the LLM's performance on general tasks. We hope that our benchmark and defenses can inspire future work in this important area.

翻译：近期，大型语言模型（LLMs）取得了显著进展，并广泛应用于各类应用程序。此类应用的核心特征是将LLMs与外部内容相结合，即通过融合用户指令与第三方内容构建提示词，供LLM处理。然而，这类应用易遭受间接提示注入攻击：嵌入在外部内容中的恶意指令会破坏LLM的输出，导致其响应偏离用户预期。尽管该安全问题已被发现，但由于缺少基准测试，目前尚无针对不同LLMs的间接提示注入攻击的综合分析，也尚未提出有效的防御方法。本研究首次提出基准测试BIPIA，用于评估各类LLMs及防御方法对间接提示注入攻击的鲁棒性。实验结果表明，在文本任务中，能力更强的LLMs对间接提示注入攻击的脆弱性更高，表现为更高的攻击成功率（ASR）。我们推测，间接提示注入攻击的主要原因是LLMs无法区分指令与外部内容。基于此假设，我们提出了四种基于提示学习的黑盒防御方法和一种基于对抗训练微调的白盒防御方法，使LLMs能够区分指令与外部内容，并忽略外部内容中的指令。实验结果显示，黑盒防御方法可有效降低ASR，但无法完全阻止间接提示注入攻击；而白盒防御方法能将ASR降至接近零，且对LLM在通用任务上的性能影响极小。我们希望所提出的基准测试与防御方法能为这一重要领域的未来研究提供启发。