Prepending model inputs with safety prompts is a common practice for safeguarding large language models (LLMs) against queries with harmful intents. However, the underlying working mechanisms of safety prompts have not been unraveled yet, restricting the possibility of automatically optimizing them to improve LLM safety. In this work, we investigate how LLMs' behavior (i.e., complying with or refusing user queries) is affected by safety prompts from the perspective of model representation. We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction, in which models become more prone to refusing to provide assistance, even when the queries are harmless. On the other hand, LLMs are naturally capable of distinguishing harmful and harmless queries without safety prompts. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO (Directed Representation Optimization). Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness. Experiments with eight LLMs on out-of-domain and jailbreak benchmarks demonstrate that DRO remarkably improves the safeguarding performance of human-crafted safety prompts, without compromising the models' general performance.

翻译：在大型语言模型（LLMs）输入前添加安全提示是防止模型响应有害意图查询的常见做法。然而，安全提示的内在工作机制尚未被充分揭示，这限制了通过自动优化提示以提升LLM安全性的可能性。本研究从模型表征的视角，探究安全提示如何影响LLMs的行为（即服从或拒绝用户查询）。我们发现，在表征空间中，安全提示通常会将输入查询向"更高拒绝倾向"的方向移动，使得模型更倾向于拒绝提供协助——即使面对无害查询时亦是如此。另一方面，LLMs在没有安全提示的情况下天然具备区分有害与无害查询的能力。受这些发现启发，我们提出了一种安全提示优化方法——定向表征优化（DRO）。该方法将安全提示视为连续可训练的嵌入向量，通过学习使查询表征根据其有害性沿着或逆着拒绝方向移动。在八个LLM上进行的跨领域和越狱基准测试表明，DRO显著提升了人工设计安全提示的防护性能，且未损害模型的通用能力。