As a data-driven paradigm, offline reinforcement learning (RL) has been formulated as sequence modeling that conditions on the hindsight information including returns, goal or future trajectory. Although promising, this supervised paradigm overlooks the core objective of RL that maximizes the return. This overlook directly leads to the lack of trajectory stitching capability that affects the sequence model learning from sub-optimal data. In this work, we introduce the concept of max-return sequence modeling which integrates the goal of maximizing returns into existing sequence models. We propose Reinforced Transformer (Reinformer), indicating the sequence model is reinforced by the RL objective. Reinformer additionally incorporates the objective of maximizing returns in the training phase, aiming to predict the maximum future return within the distribution. During inference, this in-distribution maximum return will guide the selection of optimal actions. Empirically, Reinformer is competitive with classical RL methods on the D4RL benchmark and outperforms state-of-the-art sequence model particularly in trajectory stitching ability. Code is public at \url{https://github.com/Dragon-Zhuang/Reinformer}.

翻译：作为数据驱动范式，离线强化学习（RL）已被形式化为基于后验信息（包括回报、目标或未来轨迹）进行条件限定的序列建模。尽管前景可观，这种监督范式却忽视了强化学习的核心目标——最大化回报。这种忽视直接导致缺乏轨迹拼接能力，从而影响序列模型从次优数据中学习的效果。本文提出最大回报序列建模概念，将最大化回报的目标融入现有序列模型。我们提出强化Transformer（Reinformer），表明该序列模型通过RL目标得到增强。Reinformer在训练阶段额外融合了最大化回报的目标，旨在预测分布内的最大未来回报。推理时，这种分布内最大回报将引导最优动作的选择。实验表明，Reinformer在D4RL基准上与经典RL方法具有竞争力，尤其在轨迹拼接能力上超越现有最先进的序列模型。代码已开源至 \url{https://github.com/Dragon-Zhuang/Reinformer}。