With the rapid development of large language models (LLMs) and ever-evolving practical requirements, finding an efficient and effective alignment method has never been more critical. However, the tension between the complexity of current alignment methods and the need for rapid iteration in deployment scenarios necessitates the development of a model-agnostic alignment approach that can operate under these constraints. In this paper, we introduce Aligner, a novel and simple alignment paradigm that learns the correctional residuals between preferred and dispreferred answers using a small model. Designed as a model-agnostic, plug-and-play module, Aligner can be directly applied to various open-source and API-based models with only one-off training, making it suitable for rapid iteration. Notably, Aligner can be applied to any powerful, large-scale upstream models. Moreover, it can even iteratively bootstrap the upstream models using corrected responses as synthetic human preference data, breaking through the model's performance ceiling. Our experiments demonstrate performance improvements by deploying the same Aligner model across 11 different LLMs, evaluated on the 3H dimensions (helpfulness, harmlessness, and honesty). Specifically, Aligner-7B has achieved an average improvement of 68.9\% in helpfulness and 23.8\% in harmlessness across the tested LLMs while also effectively reducing hallucination. In the Alpaca-Eval leaderboard, stacking Aligner-2B on GPT-4 Turbo improved its LC Win Rate from 55.0\% to 58.3\%, surpassing GPT-4 Omni's 57.5\% Win Rate (community report).

翻译：随着大语言模型（LLM）的快速发展与实际需求的不断演进，寻求一种高效且有效的对齐方法变得前所未有的重要。然而，当前对齐方法的复杂性与部署场景中快速迭代需求之间的张力，亟需开发一种模型无关的对齐方法以应对这些约束。本文提出对齐器（Aligner），一种新颖而简洁的对齐范式，它通过一个小型模型学习偏好答案与非偏好答案之间的修正残差。该模块被设计为模型无关、即插即用的组件，仅需一次性训练即可直接应用于各类开源模型及基于API的模型，适用于快速迭代场景。值得注意的是，对齐器可应用于任何强大的大规模上游模型。此外，它甚至能使用修正后的响应作为合成人类偏好数据，通过迭代自举提升上游模型性能，从而突破模型的能力上限。我们在11种不同LLM上部署同一对齐器模型的实验表明，其在3H维度（有益性、无害性、诚实性）上均实现了性能提升。具体而言，Aligner-7B在所有测试LLM中有益性平均提升68.9%，无害性平均提升23.8%，同时有效减少了幻觉现象。在Alpaca-Eval排行榜中，将Aligner-2B叠加于GPT-4 Turbo上，其LC胜率从55.0%提升至58.3%，超越了GPT-4 Omni报告的57.5%胜率（社区报告）。