Direct Preference Optimization (DPO) and its variants have become standard for aligning Large Language Models due to their simplicity and offline stability. However, we identify two fundamental limitations. First, the optimal policy depends on arbitrary modeling choices (scalarization function, reference policy), yielding behavior reflecting parameterization artifacts rather than true preferences. Second, treating response generation in isolation fails to leverage comparative information in pairwise data, leaving the model's capacity for intrinsic self-reflection untapped. To address it, we propose Intrinsic Self-reflective Preference Optimization (InSPO), deriving a globally optimal policy conditioning on both context and alternative responses. We prove this formulation superior to DPO/RLHF while guaranteeing invariance to scalarization and reference choices. InSPO serves as a plug-and-play enhancement without architectural changes or inference overhead. Experiments demonstrate consistent improvements in win rates and length-controlled metrics, validating that unlocking self-reflection yields more robust, human-aligned LLMs. Our Code is available at https://github.com/Skylanding/InSPO.

翻译：直接偏好优化及其变体因其简单性和离线稳定性，已成为对齐大语言模型的标准方法。然而，我们发现了两个根本性局限。首先，最优策略依赖于任意的建模选择，导致其行为反映的是参数化伪影而非真实偏好。其次，孤立地处理响应生成未能利用成对数据中的比较信息，使得模型内在的自反思能力未被开发。为此，我们提出内在自反思偏好优化，推导出一个以上下文和备选响应为条件的全局最优策略。我们证明该公式优于直接偏好优化，同时保证了对标量化函数和参考策略选择的不变性。该优化方法可作为即插即用的增强模块，无需改变架构或增加推理开销。实验结果表明，其在胜率和长度控制指标上均取得持续提升，验证了解锁自反思能力能够产生更稳健、更符合人类价值观的大语言模型。我们的代码已公开于 https://github.com/Skylanding/InSPO。