Prepending model inputs with safety prompts is a common practice for safeguarding large language models (LLMs) against queries with harmful intents. However, the underlying working mechanisms of safety prompts have not been unraveled yet, restricting the possibility of automatically optimizing them to improve LLM safety. In this work, we investigate how LLMs' behavior (i.e., complying with or refusing user queries) is affected by safety prompts from the perspective of model representation. We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction, in which models become more prone to refusing to provide assistance, even when the queries are harmless. On the other hand, LLMs are naturally capable of distinguishing harmful and harmless queries without safety prompts. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO (Directed Representation Optimization). Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness. Experiments with eight LLMs on out-of-domain and jailbreak benchmarks demonstrate that DRO remarkably improves the safeguarding performance of human-crafted safety prompts, without compromising the models' general performance.

翻译：摘要：在模型输入前添加安全提示是保护大型语言模型免受恶意意图查询攻击的常见做法。然而，安全提示的底层工作机制尚未被揭示，这限制了通过自动优化安全提示以提升模型安全性的可能性。本研究从模型表示的角度探究了安全提示如何影响大型语言模型的行为（即服从或拒绝用户查询）。我们发现，在表示空间中，安全提示通常会将输入查询向"高拒绝"方向移动，使得模型更倾向于拒绝提供帮助，即使面对无害查询也是如此。另一方面，大型语言模型本身具备区分有害与无害查询的能力，无需借助安全提示。受这些发现启发，我们提出了一种名为定向表示优化的安全提示优化方法。该方法将安全提示视为连续可训练的嵌入向量，根据查询的有害性，学习将查询的表示沿拒绝方向或相反方向移动。在八个大型语言模型上进行的跨领域及越狱攻击基准测试结果表明，定向表示优化能够显著提升人工设计安全提示的保护性能，同时不损害模型的通用能力。