We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tune large foundation models using reinforcement learning (RL), and the multi-dimensionality, heterogeneity, and conflicting nature of human preferences further complicate the alignment process. In this paper, we introduce Rewards-in-Context (RiC), which conditions the response of a foundation model on multiple rewards in its prompt context and applies supervised fine-tuning for alignment. The salient features of RiC are simplicity and adaptivity, as it only requires supervised fine-tuning of a single foundation model and supports dynamic adjustment for user preferences during inference time. Inspired by the analytical solution of an abstracted convex optimization problem, our dynamic inference-time adjustment method approaches the Pareto-optimal solution for multiple objectives. Empirical evidence demonstrates the efficacy of our method in aligning both Large Language Models (LLMs) and diffusion models to accommodate diverse rewards with only around 10% GPU hours compared with multi-objective RL baseline.

翻译：我们研究了基础模型与人类偏好多目标对齐的问题，这是构建有益且无害人工智能系统的关键步骤。然而，使用强化学习对大型基础模型进行微调通常成本高昂且不稳定，而人类偏好的多维性、异质性和冲突性进一步加剧了对齐过程的复杂性。本文提出基于上下文奖励的方法，该方法将基础模型的响应条件置于提示上下文的多个奖励信号上，并应用监督式微调进行对齐。RiC的核心特点是简洁性与自适应性：仅需对单一基础模型进行监督式微调，并在推理阶段支持用户偏好的动态调整。受抽象凸优化问题解析解的启发，我们的动态推理时调整方法能够逼近多目标问题的帕累托最优解。实证结果表明，我们的方法在对齐大型语言模型和扩散模型以适应多样化奖励方面具有显著效果，且所需GPU计算时间仅为多目标强化学习基准方法的约10%。