Learning from human preferences is important for language models to be helpful and useful for humans, and to align with human and social values. Existing works focus on supervised finetuning of pretrained models, based on curated model generations that are preferred by human labelers. Such works have achieved remarkable successes in understanding and following instructions (e.g., InstructGPT, ChatGPT, etc). However, to date, a key limitation of supervised finetuning is that it cannot learn from negative ratings; models are only trained on positive-rated data, which makes it data inefficient. Because collecting human feedback data is both time consuming and expensive, it is vital for the model to learn from all feedback, akin to the remarkable ability of humans to learn from diverse feedback. In this work, we propose a novel technique called Hindsight Finetuning for making language models learn from diverse human feedback. In fact, our idea is motivated by how humans learn from hindsight experience. We condition the model on a sequence of model generations paired with hindsight feedback, and finetune the model to predict the most preferred output. By doing so, models can learn to identify and correct negative attributes or errors. Applying the method to GPT-J, we observe that it significantly improves results on summarization and dialogue tasks using the same amount of human feedback.

翻译：学习人类偏好对于语言模型在帮助和服务人类、与人类社会价值观保持一致方面至关重要。现有工作侧重于基于经过人类标注者优选的人工生成内容，对预训练模型进行监督微调。这类方法在理解与遵循指令方面取得了显著成功（例如InstructGPT、ChatGPT等）。然而，至今监督微调的一个关键局限在于其无法从负面评价中学习——模型仅基于正面评价数据进行训练，导致数据效率低下。由于收集人类反馈数据既耗时又昂贵，因此让模型像人类那样从多样化反馈中学习至关重要。本研究提出一种名为"后见微调"的新技术，使语言模型能够从多样化的人类反馈中学习。实际上，我们的灵感来源于人类如何从经验反思中学习。我们将模型置于"模型生成序列-后见反馈"的配对条件中，通过微调使模型预测最受偏好的输出。通过这种方式，模型能够学会识别并修正负面属性或错误。将本方法应用于GPT-J模型后，观察到在使用相同数量人类反馈的情况下，该模型在摘要和对话任务上的表现显著提升。