We present BlenderBot 3x, an update on the conversational model BlenderBot 3, which is now trained using organic conversation and feedback data from participating users of the system in order to improve both its skills and safety. We are publicly releasing the participating de-identified interaction data for use by the research community, in order to spur further progress. Training models with organic data is challenging because interactions with people "in the wild" include both high quality conversations and feedback, as well as adversarial and toxic behavior. We study techniques that enable learning from helpful teachers while avoiding learning from people who are trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is both preferred in conversation to BlenderBot 3, and is shown to produce safer responses in challenging situations. While our current models are still far from perfect, we believe further improvement can be achieved by continued use of the techniques explored in this work.

翻译：我们提出BlenderBot 3x，这是对话模型BlenderBot 3的更新版本，现利用参与系统用户的有机对话与反馈数据进行训练，以提升其技能和安全性。我们公开发布了参与者的去标识化交互数据，供研究社区使用，以推动进一步进展。使用有机数据训练模型颇具挑战性，因为"野外"与人类的交互既包含高质量对话和反馈，也包含对抗性和毒性行为。我们研究了相关技术，能够从有益的教师那里学习，同时避免从试图诱导模型产生无益或毒性响应的用户处学习。BlenderBot 3x在对话中比BlenderBot 3更受欢迎，且在困难情境中能产生更安全的响应。尽管当前模型仍远未完美，但我们相信通过持续应用本工作中探索的技术，可以取得进一步改进。