Preventing jailbreaking and model hijacking of Large Language Models (LLMs) is an important yet challenging task. When interacting with a chatbot, malicious users can input specially crafted prompts that cause the LLM to generate undesirable content or perform a different task from its intended purpose. Existing systems attempt to mitigate this by hardening the LLM's system prompt or using additional classifiers to detect undesirable content or off-topic conversations. However, these probabilistic approaches are relatively easy to bypass due to the very large space of possible inputs and undesirable outputs. We present and evaluate Highlight & Summarize (H&S), a new design pattern for retrieval-augmented generation (RAG) systems that prevents these attacks by design. The core idea is to perform the same task as a standard RAG pipeline (i.e., to provide natural language answers to questions, based on relevant sources) without ever revealing the user's question to the generative LLM. This is achieved by splitting the pipeline into two components: a highlighter, which takes the user's question and extracts ("highlights") relevant passages from the retrieved documents, and a summarizer, which takes the highlighted passages and summarizes them into a cohesive answer. We describe and implement several possible instantiations of H&S and evaluate their responses in terms of correctness, relevance, and quality. For certain question-answering (QA) tasks, the responses produced by H&S are judged to be as good, if not better, than those of a standard RAG pipeline.

翻译：防止大型语言模型（LLM）的越狱和模型劫持是一项重要但具有挑战性的任务。在与聊天机器人交互时，恶意用户可输入精心构造的提示，导致LLM生成不良内容或执行与其预期目的不同的任务。现有系统尝试通过强化LLM的系统提示或使用额外分类器来检测不良内容或离题对话以缓解此问题。然而，由于可能输入和不良输出的空间极其庞大，这些概率性方法相对容易被绕过。我们提出并评估了“突出与总结”（H&S）——一种新的检索增强生成（RAG）系统设计模式，该模式从设计上防止了此类攻击。其核心思想是执行与标准RAG流程相同的任务（即基于相关来源以自然语言回答问题），同时永不向生成式LLM暴露用户的问题。这是通过将流程拆分为两个组件实现的：一个突出器，接收用户问题并从检索到的文档中提取（“突出”）相关段落；以及一个总结器，接收突出段落并将其总结为连贯的答案。我们描述并实现了H&S的几种可能实例化方案，并从正确性、相关性和质量方面评估了它们的响应。对于某些问答（QA）任务，H&S生成的响应被判定为与标准RAG流程的响应相当甚至更优。