We introduce Robust Multi-Objective Decoding (RMOD), a novel inference-time algorithm that robustly aligns Large Language Models (LLMs) to multiple human objectives (e.g., instruction-following, helpfulness, safety) by maximizing the worst-case rewards. RMOD formulates the robust decoding problem as a maximin two-player game between adversarially computed reward weights and the sampling policy, solvable through a Nash equilibrium. We demonstrate that this game reduces to a convex optimization problem to identify the worst-case reward weights, with the optimal sampling policy analytically derived. For practical applications, we propose an efficient algorithm of RMOD tailored for contemporary LLMs, introducing minimal computational overhead compared to standard non-robust Controlled Decoding methods. Experimental results across a range of popular alignment datasets with up to 10 objectives show the effectiveness of RMOD and its distilled version, consistently outperforming baselines in worst-case rewards and win rates.

翻译：我们提出了鲁棒多目标解码（RMOD），一种新颖的推理时算法，通过最大化最坏情况奖励，将大型语言模型（LLMs）鲁棒地对齐到多个人类目标（例如，指令遵循、帮助性、安全性）。RMOD将鲁棒解码问题表述为对抗性计算的奖励权重与采样策略之间的极大极小二人博弈，可通过纳什均衡求解。我们证明该博弈可简化为一个凸优化问题以识别最坏情况奖励权重，并解析推导出最优采样策略。针对实际应用，我们提出了一种为当代LLMs定制的高效RMOD算法，与标准的非鲁棒受控解码方法相比，仅引入极小的计算开销。在多达10个目标的一系列流行对齐数据集上的实验结果表明，RMOD及其蒸馏版本均有效，在最坏情况奖励和胜率方面持续优于基线方法。