An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback increase the LLM's proneness to being prompted into the undesired behaviors. Moreover, we include the notion of personas in our BEB framework, and find that behaviors which are generally very unlikely to be exhibited by the model can be brought to the front by prompting the model to behave as specific persona. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

翻译：开发与人类交互的语言模型时，一个重要方面是使其行为对人类用户有用且无害。这通常通过调整模型以增强期望行为并抑制非期望行为来实现，这一过程被称为对齐。在本文中，我们提出了一种名为行为期望界（BEB）的理论方法，使我们能够正式研究大型语言模型对齐的若干固有特征和限制。重要的是，我们证明：对于模型可能展示的任何具有有限概率的行为，存在可以触发模型输出该行为的提示，且触发概率随提示长度增加而增加。这意味着任何衰减非期望行为但未完全消除该行为的对齐过程，都无法抵御对抗性提示攻击。此外，我们的框架揭示了诸如基于人类反馈的强化学习等主流对齐方法为何会加剧LLM被诱导输出非期望行为的倾向。进一步地，我们将角色概念纳入BEB框架，发现通常极不可能被模型展现的行为，可以通过提示模型扮演特定角色而被激活。这一理论结果在所谓当代"ChatGPT越狱"攻击中得到了大规模实验验证——对抗用户通过触发LLM扮演恶意角色，诱使其突破对齐护栏。我们的研究揭示了LLM对齐的根本性局限，凸显了设计可靠机制以确保AI安全的迫切需求。