Offline reinforcement learning (RL) is a variant of RL where the policy is learned from a previously collected dataset of trajectories and rewards. In our work, we propose a practical approach to offline RL with large language models (LLMs). We recast the problem as reward-weighted fine-tuning, which can be solved using similar techniques to supervised fine-tuning (SFT). To showcase the value of our approach, we apply it to learning short-horizon question-answering policies of a fixed length, where the agent reasons about potential answers or asks clarifying questions. Our work stands in a stark contrast to state-of-the-art methods in this domain, based on SFT and direct preference optimization, which have additional hyper-parameters and do not directly optimize for rewards. We compare to them empirically, and report major gains in both optimized rewards and language quality.

翻译：离线强化学习（RL）是强化学习的一种变体，其策略从先前收集的轨迹与奖励数据集中学习。在本研究中，我们提出了一种针对大型语言模型（LLM）的实用离线强化学习方法。我们将该问题重新定义为奖励加权微调问题，该问题可采用与监督微调（SFT）相似的技术求解。为展示本方法的实用价值，我们将其应用于学习固定长度的短时域问答策略，其中智能体对潜在答案进行推理或提出澄清性问题。本研究与该领域基于监督微调与直接偏好优化的前沿方法形成鲜明对比——这些方法存在额外超参数且不直接优化奖励函数。我们通过实证比较，在优化奖励与语言质量两方面均取得了显著提升。