We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tune large foundation models using reinforcement learning (RL), and the multi-dimensionality, heterogeneity, and conflicting nature of human preferences further complicate the alignment process. In this paper, we introduce Rewards-in-Context (RiC), which conditions the response of a foundation model on multiple rewards in its prompt context and applies supervised fine-tuning for alignment. The salient features of RiC are simplicity and adaptivity, as it only requires supervised fine-tuning of a single foundation model and supports dynamic adjustment for user preferences during inference time. Inspired by the analytical solution of an abstracted convex optimization problem, our dynamic inference-time adjustment method approaches the Pareto-optimal solution for multiple objectives. Empirical evidence demonstrates the efficacy of our method in aligning both Large Language Models (LLMs) and diffusion models to accommodate diverse rewards with only around 10% GPU hours compared with multi-objective RL baseline.

翻译：本文研究基础模型与人类偏好进行多目标对齐的问题，这是构建有益且无害的人工智能系统的关键步骤。然而，使用强化学习微调大型基础模型通常成本高昂且不稳定，而人类偏好的多维性、异质性和冲突性进一步复杂化了对齐过程。为此，我们提出奖励情境方法，该方法将基础模型的响应条件设定为其提示情境中的多重奖励，并采用监督式微调实现对齐。RiC的核心特征在于简洁性与适应性：其仅需对单一基础模型进行监督式微调，即可在推理阶段支持用户偏好的动态调整。受抽象凸优化问题解析解的启发，我们提出的动态推理时调整方法可逼近多目标的帕累托最优解。实验证据表明，该方法在将大型语言模型和扩散模型对齐至多样化奖励时具有显著效果，相较于多目标强化学习基线方法，仅需约10%的GPU算力消耗。