Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. % , or Meta's LLaMA-2. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e.~supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.

翻译：通过人类反馈的强化学习（RLHF）微调的大型语言模型（LLM）已应用于目前最广泛部署的AI模型中，例如OpenAI的ChatGPT或Anthropic的Claude。尽管已有大量工作开发这些方法，但我们对于RLHF各阶段利弊的理解仍十分有限。为填补这一空白，我们系统分析了该流程的每个阶段（即有监督微调（SFT）、奖励建模和RLHF）如何影响两个关键特性：分布外（OOD）泛化能力和输出多样性。鉴于这些模型被广泛应用于各类真实场景，OOD泛化能力至关重要；而输出多样性则指模型生成多样化输出的能力，这对多种应用场景具有重要意义。我们基于两个基础模型，在摘要生成和指令遵循任务上展开分析——后者与当前LLM应用高度相关。研究发现：与SFT相比，RLHF对新型输入具有更强的泛化能力，尤其当训练与测试之间的分布偏移增大时表现更显著；然而在多种评测指标下，RLHF显著降低了输出多样性，这表明当前LLM微调方法在泛化性与多样性之间存在权衡。我们的研究结果为基于应用场景选择微调方法提供了指导，同时表明需要更多研究来改善泛化性与多样性之间的平衡。