Learning from human preferences is important for language models to be helpful and useful for humans, and to align with human and social values. Prior work have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them ineffective in terms of data utilization and challenging to apply in general, or they depend on reward functions and reinforcement learning, which are prone to imperfect reward function and extremely challenging to optimize. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, models are trained to generate outputs based on feedback, and models can learn to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We observed significant improvements on summarization and dialogue tasks and our approach is markedly preferred in human evaluations.

翻译：从人类偏好中学习对于语言模型更好地服务人类、遵循人类与社会价值观至关重要。先前的研究通过从人类反馈中学习理解与执行指令已取得显著成果。然而，这些方法要么依赖人类标注者人工筛选的模型生成结果，导致数据利用率低且难以泛化应用；要么依赖奖励函数与强化学习，而后者易受不完美奖励函数的影响且优化极为困难。本文提出一种名为"Chain of Hindsight"的新型方法，该方法易于优化，并能从任意极性反馈中学习。我们的灵感来源于人类如何通过语言形式呈现的广泛反馈进行学习。我们将所有类型的反馈转化为句子，用于模型微调，从而充分利用语言模型的语言理解能力。通过将模型条件设置为模型生成序列与对应反馈的配对，模型能够基于反馈生成输出，并学习识别与修正负面属性或错误。将该方法应用于大语言模型后，我们观察到Chain of Hindsight在使语言模型与人类偏好对齐方面显著超越先前方法。在文本摘要与对话任务中，我们观察到显著改进，且人类评估结果显示该方法具有明显优势。