Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.

翻译：大型语言模型对齐被广泛应用于避免产生无益和有害的回应。然而，冗长的训练过程和预设的偏好偏差阻碍了模型适应在线环境中多样的人类偏好。为此，本文提出一种名为“基于人类行为的强化学习”（RLHB）的对齐框架，通过直接利用真实在线人类行为来对齐大型语言模型。采用生成对抗框架，生成器被训练以遵循预期人类行为进行回应；而判别器则试图验证查询、回应和人类行为三元组是否来自真实在线环境。自然语言形式的行为建模与多模型联合训练机制实现了主动且可持续的在线对齐。人工评估和自动评估的实验结果均证实了我们所提出方法的有效性。