Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training, and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: by injecting a small amount of poisonous data (1-5% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.

翻译：从人类反馈中强化学习（RLHF）是使语言模型（LM）与人类价值观和偏好对齐的流行方法。RLHF需要大量偏好对作为训练数据，这些数据通常用于监督微调和奖励模型训练，因此公开数据集被广泛使用。本研究探讨了恶意行为者如何通过污染偏好（即向这些数据集和RLHF训练过程中注入恶意偏好对）来操纵LM的生成。我们提出了构建恶意偏好对的策略，并通过污染两个广泛使用的偏好数据集测试其性能。结果表明，偏好污染极其有效：通过注入少量恶意数据（占原始数据集的1-5%），我们能够有效操纵LM以目标情感（正面或负面）生成目标实体。实验发现也为防御偏好污染攻击的策略提供了启示。