The widespread application of large language models (LLMs) raises increasing demands on ensuring safety or imposing constraints, such as reducing harmful content and adhering to predefined rules. While there have been several works studying LLM safety alignment, these works either need to train three models and incur high memory costs, or require prior knowledge on the optimal solution. Witnessing this fact, we investigate the constrained alignment problem for LLMs, i.e., maximizing the reward of outputs while restricting the cost to stay below a threshold. We propose a novel primal-dual direct preference optimization (DPO) approach, which first trains a model using standard DPO on reward preference data to provide reward information, and then adopts a rearranged Lagrangian DPO objective utilizing the provided reward information to fine-tune LLMs. Our approach only needs to train two models rather than three, which significantly saves memory costs, and does not require extra prior knowledge. Moreover, we establish rigorous suboptimality and constraint violation guarantees. We also extend our approach to enable online exploration and drop the data coverage dependence in the results. Experiments on the PKU-SafeRLHF and TruthfulQA datasets demonstrate the state-of-the-art performance of our approach.

翻译：大型语言模型(LLM)的广泛应用对确保安全性或施加约束(如减少有害内容和遵守预定义规则)提出了日益增长的需求。尽管已有若干研究关注LLM安全对齐问题，但这些方法要么需要训练三个模型并产生高昂内存开销，要么需要关于最优解的先验知识。基于此现状，我们研究LLM的约束对齐问题，即在限制成本低于阈值的前提下最大化输出奖励。我们提出一种新颖的原始-对偶直接偏好优化(DPO)方法：首先使用标准DPO在奖励偏好数据上训练模型以提供奖励信息，随后采用重构的拉格朗日DPO目标函数，利用已获得的奖励信息对LLM进行微调。该方法仅需训练两个模型而非三个，显著节约内存成本，且无需额外先验知识。此外，我们建立了严格的次优性与约束违反保证理论。我们还将该方法扩展至支持在线探索，并消除了结果中对数据覆盖度的依赖。在PKU-SafeRLHF和TruthfulQA数据集上的实验验证了本方法的先进性能。