Adversarial training (AT) is an effective defense for large language models (LLMs) against jailbreak attacks, but performing AT on LLMs is costly. To improve the efficiency of AT for LLMs, recent studies propose continuous AT (CAT) that searches for adversarial inputs within the continuous embedding space of LLMs during AT. While CAT has achieved empirical success, its underlying mechanism, i.e., why adversarial perturbations in the embedding space can help LLMs defend against jailbreak prompts synthesized in the input token space, remains unknown. This paper presents the first theoretical analysis of CAT on LLMs based on in-context learning (ICL) theory. For linear transformers trained with adversarial examples from the embedding space on in-context linear regression tasks, we prove a robust generalization bound that has a negative correlation with the perturbation radius in the embedding space. This clearly explains why CAT can defend against jailbreak prompts from the LLM's token space. Further, the robust bound shows that the robustness of an adversarially trained LLM is closely related to the singular values of its embedding matrix. Based on this, we propose to improve LLM CAT by introducing an additional regularization term, which depends on singular values of the LLM's embedding matrix, into the objective function of CAT. Experiments on real-world LLMs demonstrate that our method can help LLMs achieve a better jailbreak robustness-utility tradeoff. The code is available at https://github.com/fshp971/continuous-adv-icl.

翻译：对抗训练（AT）是大语言模型（LLMs）抵御越狱攻击的有效防御手段，但直接对LLMs进行AT成本高昂。为提高LLMs对抗训练的效率，近期研究提出了连续对抗训练（CAT），该方法在对抗训练过程中于LLMs的连续嵌入空间中搜索对抗输入。尽管CAT已取得经验性成功，但其内在机理——为何嵌入空间的对抗扰动能帮助LLMs抵御输入词元空间合成的越狱提示——仍不明确。本文基于上下文学习（ICL）理论，首次对LLMs的CAT进行了理论分析。针对在上下文线性回归任务中使用嵌入空间对抗样本训练的线性Transformer，我们证明了一个鲁棒泛化界，该界与嵌入空间中的扰动半径呈负相关。这清晰解释了CAT为何能防御来自LLMs词元空间的越狱提示。此外，该鲁棒泛化界表明，经过对抗训练的LLMs的鲁棒性与其嵌入矩阵的奇异值密切相关。基于此，我们提出通过在CAT目标函数中引入一个依赖于LLMs嵌入矩阵奇异值的额外正则化项来改进CAT。在真实LLMs上的实验表明，我们的方法能帮助LLMs实现更好的越狱鲁棒性-效用权衡。代码已开源：https://github.com/fshp971/continuous-adv-icl。