Current methods for large language model alignment typically use scalar human preference labels. However, this convention tends to oversimplify the multi-dimensional and heterogeneous nature of human preferences, leading to reduced expressivity and even misalignment. This paper presents Panacea, an innovative approach that reframes alignment as a multi-dimensional preference optimization problem. Panacea trains a single model capable of adapting online and Pareto-optimally to diverse sets of preferences without the need for further tuning. A major challenge here is using a low-dimensional preference vector to guide the model's behavior, despite it being governed by an overwhelmingly large number of parameters. To address this, Panacea is designed to use singular value decomposition (SVD)-based low-rank adaptation, which allows the preference vector to be simply injected online as singular values. Theoretically, we prove that Panacea recovers the entire Pareto front with common loss aggregation methods under mild conditions. Moreover, our experiments demonstrate, for the first time, the feasibility of aligning a single LLM to represent an exponentially vast spectrum of human preferences through various optimization methods. Our work marks a step forward in effectively and efficiently aligning models to diverse and intricate human preferences in a controllable and Pareto-optimal manner.

翻译：当前的大语言模型对齐方法通常使用标量形式的人类偏好标签。然而，这种做法往往过度简化了人类偏好多维度、异质性的本质，导致表达力下降甚至产生未对齐现象。本文提出Panacea，一种创新性方法，将对齐问题重新定义为多维度偏好优化问题。Panacea训练单一模型，使其能够在线自适应并帕累托最优地适应不同偏好集合，而无需进一步调优。此处的核心挑战在于：尽管模型行为由海量参数控制，但需使用低维偏好向量来引导模型行为。为解决此问题，Panacea设计采用基于奇异值分解（SVD）的低秩自适应方法，使得偏好向量能够以奇异值形式在线直接注入。理论上，我们证明在温和条件下，Panacea能够通过常见的损失聚合方法恢复完整的帕累托前沿。此外，我们的实验首次证明，通过多种优化方法，对齐单一LLM以表征指数级广阔的人类偏好谱系具有可行性。本工作标志着在可控且帕累托最优的方式下，高效对齐模型以适应多样化、复杂化人类偏好的研究迈出了重要一步。