An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

翻译：开发与人类交互的语言模型的一个重要方面是将其行为对齐，使其对人类用户有用且无害。这通常通过调整模型以增强期望行为并抑制非期望行为来实现，这一过程称为对齐。本文提出一种称为行为期望边界（BEB）的理论方法，使我们能够形式化地研究大语言模型对齐的若干固有特性与局限性。重要的是，我们证明在该框架的限定范围内，对于模型具有有限概率表现出的任何行为，都存在能触发模型输出该行为的提示，且其概率随提示长度增加而提升。这意味着任何仅衰减非期望行为而未完全消除的对齐过程，都无法抵御对抗性提示攻击。此外，我们的框架揭示了基于人类反馈的强化学习等主流对齐方法使LLM易受提示诱导而表现非期望行为的内在机制。这一理论结果正通过当代所谓的"ChatGPT越狱"现象得到大规模实验验证——对抗性用户通过触发LLM扮演恶意角色来突破其对齐防护机制。我们的研究结果揭示了LLM对齐的根本局限性，并凸显了构建可靠人工智能安全机制的必要性。