The Reinforcement Learning from Human Feedback (RLHF) plays a pivotal role in shaping the impact of large language models (LLMs), contributing significantly to controlling output toxicity and selecting output styles, particularly as LLMs often harbor misleading content, highlighting the urgency to align them with human values for secure AI systems. The RLHF, characterized by complexity, instability, and sensitivity to hyperparameters, makes the evaluation of the reward model for complex tasks challenging, thereby further complicating the use of Proximal Policy Optimization (PPO). In this paper, we introduce a simple task designed to employ Gloden as a reward model that validates the effectiveness of PPO and inspires it, primarily explaining the task of utilizing PPO to manipulate the tokenizer length of the output generated by the model. Experiments confirm that PPO is not only effective in manipulating the output tokenizer length to a certain extent in this type of task but also exhibits facilitated training once the influence of the reward model effect is excluded, making it an exciting development.

翻译：基于人类反馈的强化学习（RLHF）在塑造大语言模型（LLMs）的影响力方面起着关键作用，显著有助于控制输出毒性及选择输出风格。尤其鉴于LLMs常包含误导性内容，突显了使其与人类价值观对齐以构建安全AI系统的紧迫性。RLHF具有复杂性、不稳定性和对超参数敏感的特点，这使得复杂任务中奖励模型的评估极具挑战，从而进一步增加了近端策略优化（PPO）的使用难度。本文引入一项简单任务，旨在以Gloden作为奖励模型验证PPO的有效性并激发其潜力，主要阐释了利用PPO操纵模型生成输出的分词器长度这一任务。实验证实，在此类任务中，PPO不仅能在一定程度上有效操纵输出分词器长度，而且在排除奖励模型效应影响后展现出更便捷的训练特性，这一发现令人振奋。