We study a persistent failure mode in multi-objective alignment for large language models (LLMs): training improves performance on only a subset of objectives while causing others to degrade. We formalize this phenomenon as cross-objective interference and conduct the first systematic study across classic scalarization algorithms, showing that interference is pervasive and exhibits strong model dependence. To explain this phenomenon, we derive a local covariance law showing that an objective improves at first order when its reward exhibits positive covariance with the scalarized score. We extend this analysis to clipped surrogate objectives used in modern alignment, demonstrating that the covariance law remains valid under mild conditions despite clipping. Building on this analysis, we propose Covariance Targeted Weight Adaptation (CTWA), a plug-and-play method that maintains positive covariance between objective rewards and the training signal to effectively mitigate cross-objective interference. Finally, we complement these local improvement conditions with a global convergence analysis under the Polyak--Łojasiewicz condition, establishing when non-convex scalarized optimization achieves global convergence and how cross-objective interference depends on specific model geometric properties.

翻译：我们研究大型语言模型（LLM）多目标对齐中一种持续存在的失效模式：训练仅提升部分目标的性能，同时导致其他目标性能下降。我们将此现象形式化为跨目标干扰，并首次对经典标量化算法进行了系统性研究，表明干扰普遍存在且表现出强烈的模型依赖性。为解释此现象，我们推导了一个局部协方差定律，证明当目标的奖励与标量化分数呈现正协方差时，该目标在一阶意义上得到改进。我们将此分析扩展到现代对齐中使用的裁剪代理目标，证明在温和条件下，尽管存在裁剪，协方差定律仍然成立。基于此分析，我们提出了协方差目标权重适配（CTWA），这是一种即插即用方法，通过保持目标奖励与训练信号之间的正协方差来有效缓解跨目标干扰。最后，我们结合Polyak–Łojasiewicz条件下的全局收敛分析，补充了这些局部改进条件，确立了非凸标量化优化何时实现全局收敛，以及跨目标干扰如何依赖于特定的模型几何特性。