Learning from human preferences is important for language models to be helpful and useful for humans, and to align with human and social values. Existing works focus on supervised finetuning of pretrained models, based on curated model generations that are preferred by human labelers. Such works have achieved remarkable successes in understanding and following instructions (e.g., InstructGPT, ChatGPT, etc). However, to date, a key limitation of supervised finetuning is that it cannot learn from negative ratings; models are only trained on positive-rated data, which makes it data inefficient. Because collecting human feedback data is both time consuming and expensive, it is vital for the model to learn from all feedback, akin to the remarkable ability of humans to learn from diverse feedback. In this work, we propose a novel technique called Hindsight Finetuning for making language models learn from diverse human feedback. In fact, our idea is motivated by how humans learn from hindsight experience. We condition the model on a sequence of model generations paired with hindsight feedback, and finetune the model to predict the most preferred output. By doing so, models can learn to identify and correct negative attributes or errors. Applying the method to GPT-J, we observe that it significantly improves results on summarization and dialogue tasks using the same amount of human feedback.

翻译：从人类偏好中学习对于语言模型提升对人类的帮助性和实用性、以及使其与人类及社会价值观对齐至关重要。现有工作主要基于人类标注员偏好的精选模型生成结果，对预训练模型进行监督微调。这类工作在理解与遵循指令（如InstructGPT、ChatGPT等）方面取得了显著成功。然而，目前监督微调的一个关键局限在于无法从负面评价中学习——模型仅接受正向评分数据的训练，导致数据利用效率低下。由于收集人类反馈数据既耗时又昂贵，让模型从各类反馈中学习至关重要，这与人类从多样化反馈中学习的卓越能力异曲同工。本文提出一种名为"后见微调"的新颖技术，使语言模型能够从多样化的人类反馈中学习。事实上，我们的灵感源于人类从后见经验中学习的方式：我们根据配对的模型生成序列与后见反馈对模型进行条件化处理，通过微调使模型预测最受偏好的输出。通过这种方式，模型能够学习识别并修正负面属性或错误。将该方法应用于GPT-J模型后，我们发现其在相同人类反馈数据量下，显著提升了摘要生成与对话任务的性能。