Bias in LLMs can harm user experience and societal outcomes. However, current bias mitigation methods often require intensive human feedback, lack transferability to other topics or yield overconfident and random outputs. We find that involving LLMs in role-playing scenario boosts their ability to recognize and mitigate biases. Based on this, we propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning. Our approach comprises two modes: (1) self-reflection, where the same LLM participates in multi-role debates, and (2) teacher-student, where a more advanced LLM like GPT-3.5-turbo guides the LLM to perform this task. Experimental results across different LLMs demonstrate the effectiveness of our approach in bias mitigation.

翻译：大语言模型中的偏见可能损害用户体验并产生不良社会影响。然而，现有的偏见缓解方法通常需要大量人工反馈、缺乏跨主题的迁移能力，或会产生过度自信及随机性输出。我们发现，让大语言模型参与角色扮演场景能有效提升其识别与缓解偏见的能力。基于此，我们提出了一种新颖的偏见缓解方法——基于多角色辩论反馈的强化学习，该方法可替代传统基于人类反馈的强化学习中的人工反馈环节。我们利用大语言模型进行多角色辩论，构建包含高偏见与低偏见实例的数据集，用于训练强化学习中的奖励模型。我们的方法包含两种模式：（1）自我反思模式：同一大语言模型参与多角色辩论；（2）师生模式：由更先进的大语言模型（如GPT-3.5-turbo）指导目标模型执行该任务。在不同大语言模型上的实验结果表明，我们的方法在偏见缓解方面具有显著有效性。