Single-trajectory reinforcement learning (RL) methods aim to optimize policies from datasets consisting of (prompt, response, reward) triplets, where scalar rewards are directly available. This supervision format is highly practical, as it mirrors real-world human feedback, such as thumbs-up/down signals, and avoids the need for structured preference annotations. In contrast, pairwise preference-based methods like Direct Preference Optimization (DPO) rely on datasets with both preferred and dispreferred responses, which are harder to construct and less natural to collect. Among single-trajectory approaches, Direct Reward Optimization (DRO) has shown strong empirical performance due to its simplicity and stability. However, DRO requires approximating a value function, which introduces several limitations: high off-policy variance, coupling between policy and value learning, and a lack of absolute supervision on the policy itself. We introduce Reward Partitioning Optimization (RPO), a new method that resolves these limitations by removing the need to model the value function. Instead, RPO normalizes observed rewards using a partitioning approach estimated directly from data. This leads to a straightforward supervised learning objective on the policy, with no auxiliary models and no joint optimization. RPO provides direct and stable supervision on the policy, making it robust and easy to implement in practice. We validate RPO on scalar-feedback language modeling tasks using Flan-T5 encoder-decoder models. Our results demonstrate that RPO outperforms existing single-trajectory baselines such as DRO and Kahneman-Tversky Optimization (KTO). These findings confirm that RPO is a simple, effective, and theoretically grounded method for single-trajectory policy optimization.

翻译：单轨迹强化学习方法旨在优化策略，其数据集由（提示、响应、奖励）三元组构成，其中标量奖励可直接获取。这种监督形式极具实用性，因为它模拟了现实世界中的人类反馈（如点赞/点踩信号），并避免了结构化偏好标注的需求。相比之下，基于成对偏好的方法（如直接偏好优化）依赖于包含偏好与非偏好响应的数据集，这类数据集的构建更为困难且收集过程更不自然。在单轨迹方法中，直接奖励优化因其简单性和稳定性而展现出强大的实证性能。然而，DRO 需要近似一个价值函数，这引入了若干局限性：高离策略方差、策略与价值学习的耦合，以及对策略本身缺乏绝对监督。我们提出了奖励划分优化，这是一种通过消除对价值函数建模需求来解决这些局限性的新方法。RPO 使用直接从数据估计的划分方法对观测到的奖励进行归一化。这产生了一个直接的策略监督学习目标，无需辅助模型，也无需联合优化。RPO 为策略提供了直接且稳定的监督，使其在实践中具有鲁棒性且易于实现。我们使用 Flan-T5 编码器-解码器模型在标量反馈语言建模任务上验证了 RPO。我们的结果表明，RPO 优于现有的单轨迹基线方法，如 DRO 和 Kahneman-Tversky 优化。这些发现证实，RPO 是一种简单、有效且理论依据充分的单轨迹策略优化方法。