We study a persistent failure mode in multi-objective alignment for large language models (LLMs): training improves performance on only a subset of objectives while causing others to degrade. We formalize this phenomenon as cross-objective interference and conduct the first systematic study across scalarization algorithms, showing that interference is pervasive and exhibits strong model dependence. To explain this phenomenon, we derive a local covariance law showing that an objective improves when its reward exhibits positive covariance with the scalarized score. We extend this analysis to clipped surrogate objectives used in modern alignment, demonstrating that the covariance law remains valid under mild conditions despite clipping. Building on this analysis, we propose Covariance Targeted Weight Adaptation (CTWA), a plug-and-play method that maintains positive covariance between objective rewards and the training signal to effectively mitigate cross-objective interference. Finally, we complement these local improvement conditions with a global convergence analysis under the Polyak--Łojasiewicz condition, establishing when non-convex scalarized optimization achieves global convergence and how cross-objective interference depends on specific model geometric properties.

翻译：我们研究了大型语言模型多目标对齐中存在的一种持续性失效模式：训练仅提升部分目标的性能，却导致其他目标退化。我们将这一现象形式化为"跨目标干扰"，并首次系统地研究了其在标量化算法中的表现，发现干扰具有普遍性且表现出显著的模型依赖性。为解释该现象，我们推导出局部协方差定律：当一个目标的奖励与标量化评分呈正协方差时，该目标的性能会提升。我们将此分析扩展至现代对齐中使用的裁剪替代目标，证明尽管存在裁剪操作，协方差定律在温和条件下依然成立。基于此分析，我们提出了协方差目标权重自适应（CTWA）方法——一种即插即用方案，通过维持目标奖励与训练信号间的正协方差来有效缓解跨目标干扰。最后，我们在Polyak--Łojasiewicz条件下补充了全局收敛性分析，阐明了非凸标量化优化实现全局收敛的条件，以及跨目标干扰如何依赖于特定的模型几何特性。