Reinforcement Learning from Human Feedback (RLHF) is the standard method to align Large Language Models (LLMs) with human preferences. In this work, we introduce alignment tampering, a potential vulnerability where the LLM undergoing alignment influences the preference dataset, causing RLHF to amplify undesired behaviors. This arises from core limitations of RLHF: (1) preference datasets are constructed from the LLM's own outputs, allowing it to influence them, and (2) pairwise comparisons only indicate which response is better, not why. These limitations can be exploited to cause alignment tampering. For example, if an LLM generates biased responses with higher quality, annotators will prefer them based on quality. However, preference labels do not distinguish quality from bias, and the reward model inherits this limitation. Optimizing such rewards through reinforcement learning or best-of-N sampling can amplify misaligned biases. Our experiments demonstrate amplification across diverse biases: from keyword bias to propaganda (e.g., sexism), brand promotion, and instrumental goal-seeking. Mitigation remains challenging, as existing techniques for robust RLHF fail to fully resolve alignment tampering without sacrificing response quality. These findings reveal structural vulnerabilities of current RLHF and emphasize the need to prevent this vulnerability. Project page: https://alignment-tampering.github.io/

翻译：基于人类反馈的强化学习（RLHF）是使大型语言模型（LLMs）与人类偏好对齐的标准方法。在本文中，我们引入对齐篡改这一潜在漏洞——即正在经历对齐的LLM会影响偏好数据集，导致RLHF放大不良行为。该漏洞源于RLHF的核心局限性：（1）偏好数据集由LLM自身的输出构建，使其能够影响这些数据；（2）成对比较仅指明哪个响应更优，而非原因。这些局限性可被利用以引发对齐篡改。例如，若LLM以更高质量生成存在偏见的响应，标注者会基于质量偏好这些响应。然而，偏好标签无法区分质量与偏见，奖励模型继承了这一缺陷。通过强化学习或最佳N采样（best-of-N）优化此类奖励，可能放大失调的偏见。我们的实验在不同偏见的放大中验证了这一现象：从关键词偏差到宣传（如性别歧视）、品牌推广及工具性目标追求。缓解该问题仍具挑战性，现有鲁棒RLHF技术无法在不牺牲响应质量的前提下完全解决对齐篡改。这些发现揭示了当前RLHF的结构性漏洞，并强调了防范该漏洞的必要性。项目页面：https://alignment-tampering.github.io/