Although the capabilities of large language models (LLMs) ideally scale up with increasing data and compute, they are inevitably constrained by limited resources in reality. Suppose we have a moderately trained LLM (e.g., trained to align with human preference) in hand, can we further exploit its potential and cheaply acquire a stronger model? In this paper, we propose a simple method called ExPO to boost LLMs' alignment with human preference. ExPO assumes that a medium-aligned model can be interpolated between a less-aligned (weaker) model, e.g., the initial SFT model, and a better-aligned (stronger) one, thereby directly obtaining this stronger model by extrapolating from the weights of the former two relatively weaker models. On the AlpacaEval 2.0 benchmark, we show that ExPO pushes models trained with less preference data (e.g., 10% or 20%) to reach and even surpass the fully-trained one, without any additional training. Furthermore, ExPO also significantly improves off-the-shelf DPO/RLHF models and exhibits decent scalability across model sizes from 7B to 70B. Our work demonstrates the efficacy of model extrapolation in exploiting LLMs' capabilities, suggesting a promising direction that deserves future exploration.

翻译：尽管大型语言模型（LLMs）的能力理想情况下会随数据和计算量的增加而扩展，但在现实中却不可避免地受到有限资源的制约。假设我们手中有一个经过适度训练的LLM（例如，为与人类偏好对齐而训练的模型），能否进一步挖掘其潜力并以低成本获得更强的模型？本文提出一种名为ExPO的简单方法，用于提升LLMs与人类偏好的对齐效果。ExPO假设，一个中等对齐的模型可以插值于一个较弱对齐模型（如初始SFT模型）和一个较好对齐模型（更强模型）之间，从而通过对前两个相对较弱模型的权重进行外推，直接获得这一更强模型。在AlpacaEval 2.0基准测试中，我们证明ExPO能使使用较少偏好数据（例如10%或20%）训练的模型达到甚至超越完全训练的模型，且无需额外训练。此外，ExPO还能显著改进现成的DPO/RLHF模型，并在7B到70B的模型规模上表现出良好的可扩展性。我们的工作证明了模型外推在挖掘LLMs能力方面的有效性，为未来值得探索的研究方向提供了启示。