Learning from human preferences is important for language models to be helpful and useful for humans, and to align with human and social values. Prior work have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them ineffective in terms of data utilization and challenging to apply in general, or they depend on reward functions and reinforcement learning, which are prone to imperfect reward function and extremely challenging to optimize. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, models are trained to generate outputs based on feedback, and models can learn to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We observed significant improvements on summarization and dialogue tasks and our approach is markedly preferred in human evaluations.

翻译：从人类偏好中学习对于语言模型变得对人类有帮助、有用，并与人类及社会价值观对齐至关重要。先前的工作通过从人类反馈中学习理解与遵循指令取得了显著成功。然而，这些方法要么基于人类标注者偏好的手工精选模型生成，导致数据利用效率低且难以广泛应用；要么依赖奖励函数与强化学习，易受不完美奖励函数影响且优化极具挑战性。本文提出一种新颖技术——链式后见之明（Chain of Hindsight），它易于优化并能从任意形式的反馈（无论极性如何）中学习。我们的灵感源于人类如何通过语言形式呈现的广泛反馈进行学习。我们将所有类型的反馈转化为句子，进而用于微调模型，从而充分利用语言模型的语言理解能力。我们使模型以模型生成序列与反馈配对的组合为条件。通过这种方式，模型被训练为基于反馈生成输出，并能够识别与纠正负面属性或错误。将我们的方法应用于大型语言模型后，观察到链式后见之明在使语言模型与人类偏好对齐方面显著超越先前方法。在摘要生成和对话任务中，我们观察到显著改进，且人工评估中我们的方法明显更受青睐。