The dominant framework for alignment of large language models (LLM), whether through reinforcement learning from human feedback or direct preference optimisation, is to learn from preference data. This involves building datasets where each element is a quadruplet composed of a prompt, two independent responses (completions of the prompt) and a human preference between the two independent responses, yielding a preferred and a dis-preferred response. Such data is typically scarce and expensive to collect. On the other hand, \emph{single-trajectory} datasets where each element is a triplet composed of a prompt, a response and a human feedback is naturally more abundant. The canonical element of such datasets is for instance an LLM's response to a user's prompt followed by a user's feedback such as a thumbs-up/down. Consequently, in this work, we propose DRO, or \emph{Direct Reward Optimisation}, as a framework and associated algorithms that do not require pairwise preferences. DRO uses a simple mean-squared objective that can be implemented in various ways. We validate our findings empirically, using T5 encoder-decoder language models, and show DRO's performance over selected baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that DRO is a simple and empirically compelling method for single-trajectory policy optimisation.

翻译：当前大语言模型（LLM）对齐的主流框架——无论是基于人类反馈的强化学习还是直接偏好优化——均依赖于从偏好数据中学习。这类数据集的每个样本通常由四元组构成：一个提示、两个独立响应（对提示的补全）以及两者间的人类偏好，从而产生一个偏好响应和一个非偏好响应。此类数据通常稀缺且收集成本高昂。相比之下，**单轨迹**数据集（每个样本由提示、响应和人类反馈组成的三元组）天然更为丰富。例如，此类数据的典型样本可包含LLM对用户提示的响应及后续用户反馈（如点赞/点踩）。为此，本研究提出DRO（**直接奖励优化**）框架及其相应算法，该框架无需成对偏好数据。DRO采用可多种方式实现的简单均方误差目标函数。我们基于T5编码器-解码器语言模型进行了实证验证，结果表明DRO在选定的基线方法（如Kahneman-Tversky优化算法）上表现优异。由此证实，DRO是一种简洁且具有实证优势的单轨迹策略优化方法。