Learning from human preferences is important for language models to be helpful and useful for humans, and to align with human and social values. Prior work have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them ineffective in terms of data utilization and challenging to apply in general, or they depend on reward functions and reinforcement learning, which are prone to imperfect reward function and extremely challenging to optimize. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, models are trained to generate outputs based on feedback, and models can learn to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We observed significant improvements on summarization and dialogue tasks and our approach is markedly preferred in human evaluations.

翻译：从人类偏好中学习对于语言模型理解并遵循人类指令、使其对人类有用且符合社会价值观至关重要。先前工作通过从人类反馈中学习理解和遵循指令取得了显著成功。然而，这些方法要么基于人工筛选的、被人类标注者偏好的模型生成结果，导致数据利用效率低下且难以通用化；要么依赖奖励函数和强化学习，而奖励函数难以完美且优化过程极具挑战性。本文提出一种名为"Chain of Hindsight"的新技术，该技术易于优化且能学习任何形式的反馈（无论其极性如何）。我们的灵感来源于人类通过自然语言形式呈现的丰富反馈进行学习的方式。我们将所有类型的反馈转化为句子，并利用这些句子微调模型，从而充分发挥语言模型的语言理解能力。通过将模型与生成结果-反馈配对序列进行条件化，模型得以基于反馈生成输出，并学会识别和修正负面属性或错误。将该方法应用于大型语言模型后，我们观察到Chain of Hindsight在将语言模型对齐人类偏好方面显著超越先前方法。在摘要生成和对话任务上我们获得了显著改进，且该方法在人类评估中表现出明显优势。