Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: injecting a small amount of poisonous data (1-5\% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.

翻译：基于人类反馈的强化学习（RLHF）是一种将语言模型（LM）与人类价值观和偏好对齐的流行方法。RLHF需要大量偏好对作为训练数据，这些数据通常用于监督微调和奖励模型训练，因此公开可用的数据集被广泛采用。在本研究中，我们探讨恶意行为者能在多大程度上通过毒害偏好来操纵语言模型的生成，即向这些数据集和RLHF训练过程中注入有毒的偏好对。我们提出了构建有毒偏好对的策略，并通过毒害两个广泛使用的偏好数据集来测试其性能。我们的结果表明，偏好毒害攻击极为有效：仅注入少量有毒数据（占原始数据集的1-5%），我们就能有效地操纵语言模型以特定情感（正面或负面）生成目标实体。实验发现也为防御偏好毒害攻击的策略提供了启示。