Learning from human feedback is a prominent technique to align the output of large language models (LLMs) with human expectations. Reinforcement learning from human feedback (RLHF) leverages human preference signals that are in the form of ranking of response pairs to perform this alignment. However, human preference on LLM outputs can come in much richer forms including natural language, which may provide detailed feedback on strengths and weaknesses of a given response. In this work we investigate data efficiency of modeling human feedback that is in natural language. Specifically, we fine-tune an open-source LLM, e.g., Falcon-40B-Instruct, on a relatively small amount (1000 records or even less) of human feedback in natural language in the form of critiques and revisions of responses. We show that this model is able to improve the quality of responses from even some of the strongest LLMs such as ChatGPT, BARD, and Vicuna, through critique and revision of those responses. For instance, through one iteration of revision of ChatGPT responses, the revised responses have 56.6% win rate over the original ones, and this win rate can be further improved to 65.9% after applying the revision for five iterations.

翻译：从人类反馈中学习是一种显著的技术，用于使大型语言模型（LLMs）的输出与人类期望对齐。基于人类反馈的强化学习（RLHF）利用以响应排序对形式呈现的人类偏好信号来实现此对齐。然而，人类对LLM输出的偏好可以以更丰富的形式呈现，包括自然语言，这可能提供对给定响应的优缺点详细反馈。在本研究中，我们探究了以自然语言形式建模人类反馈的数据效率。具体而言，我们在相对少量（1000条记录或更少）的自然语言人类反馈（以评论和响应修订形式）上微调了一个开源LLM，例如Falcon-40B-Instruct。我们证明，该模型能够通过评论和修订那些响应，提升即使是某些最强LLM（如ChatGPT、BARD和Vicuna）的响应质量。例如，通过一轮对ChatGPT响应的修订，修订后响应的胜率相比原始响应达到56.6%，而在应用五轮修订后，这一胜率可进一步提升至65.9%。