Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences. Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process. Understanding this relationship is important as real user data may be collected and used to align large models; if user data is memorized during RLHF and later regurgitated, this could raise privacy concerns. In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF. We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized, in comparison to aligning via directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF.

翻译：基于人类反馈的强化学习（RLHF）已成为使大模型与用户偏好对齐的主流方法。与微调不同（关于训练数据记忆已有大量研究），目前尚不清楚记忆效应如何受RLHF对齐过程影响或在该过程中产生。理解这种关系至关重要，因为实际用户数据可能被收集并用于对齐大模型；若用户数据在RLHF过程中被记忆并后续重现，可能引发隐私担忧。本研究分析了训练数据记忆如何通过RLHF各阶段显现和传播。我们聚焦于代码补全模型，因其是大语言模型最流行的应用场景之一。研究发现：与此类数据直接微调的对齐方式相比，RLHF显著降低了奖励建模和强化学习所用数据被记忆的概率；但已在RLHF微调阶段被记忆的示例，在绝大多数情况下经过RLHF后仍保持记忆状态。