Prepending model inputs with safety prompts is a common practice of safeguarding large language models (LLMs) from complying with queries that contain harmful intents. However, the working mechanisms of safety prompts have not yet been fully understood, which hinders the potential for automatically optimizing them for improved LLM safety. Motivated by this problem, we investigate the impact of safety prompts from the perspective of model representations. We find that in models' representation space, harmful and harmless queries can be largely distinguished, but this is not noticeably enhanced by safety prompts. Instead, the queries' representations are moved by different safety prompts in similar directions, where models become more prone to refusal (i.e., refusing to provide assistance) even when the queries are harmless. Inspired by these findings, we propose a method called DRO (Directed Representation Optimization) for automatic safety prompt optimization. DRO treats safety prompts as continuous, trainable embeddings and learns to move the representations of harmful/harmless queries along/opposite the direction in which the model's refusal probability increases. We demonstrate that DRO remarkably improves the safeguarding performance of human-crafted safety prompts and outperforms strong baselines, as evaluated on out-of-domain benchmarks, without compromising the general model capability.

翻译：在模型输入前添加安全提示是保护大语言模型（LLM）免于响应含恶意意图查询的常见做法。然而，安全提示的工作机制尚未完全明晰，这阻碍了自动优化安全提示以提升LLM安全性的潜能。针对该问题，我们从模型表示的角度探究了安全提示的影响。研究发现，在模型的表示空间中，有害与无害查询可被显著区分，但安全提示并未明显增强这一区分能力。相反，不同安全提示会将查询的表示朝相似方向移动，导致模型在无害查询场景下也更倾向于拒绝响应（即拒绝提供协助）。基于此发现，我们提出名为DRO（定向表示优化）的自动安全提示优化方法。该方法将安全提示视为连续可训练的嵌入向量，通过沿模型拒答概率增加的方向/反方向移动有害/无害查询的表示进行学习。实验表明，DRO显著提升了人工设计安全提示的防护性能，在跨领域基准测试中优于强基线方法，且未影响模型通用能力。