Recent work has shown that Large Language Models (LLMs) can be incredibly effective for offline reinforcement learning (RL) by representing the traditional RL problem as a sequence modelling problem (Chen et al., 2021; Janner et al., 2021). However many of these methods only optimize for high returns, and may not extract much information from a diverse dataset of trajectories. Generalized Decision Transformers (GDTs) (Furuta et al., 2021) have shown that utilizing future trajectory information, in the form of information statistics, can help extract more information from offline trajectory data. Building upon this, we propose Skill Decision Transformer (Skill DT). Skill DT draws inspiration from hindsight relabelling (Andrychowicz et al., 2017) and skill discovery methods to discover a diverse set of primitive behaviors, or skills. We show that Skill DT can not only perform offline state-marginal matching (SMM), but can discovery descriptive behaviors that can be easily sampled. Furthermore, we show that through purely reward-free optimization, Skill DT is still competitive with supervised offline RL approaches on the D4RL benchmark. The code and videos can be found on our project page: https://github.com/shyamsn97/skill-dt

翻译：近期研究表明，大型语言模型（LLMs）通过将传统强化学习（RL）问题重构为序列建模问题（Chen等，2021；Janner等，2021），在离线强化学习中展现出极高的有效性。然而，这些方法大多仅针对高回报进行优化，可能无法从多样化的轨迹数据集中充分提取信息。广义决策Transformer（GDTs）（Furuta等，2021）已证明，利用未来轨迹信息（以信息统计量的形式）有助于从离线轨迹数据中提取更多信息。在此基础上，我们提出技能决策Transformer（Skill DT）。Skill DT借鉴事后重标注（Andrychowicz等，2017）与技能发现方法的思想，旨在发现多样化的基本行为模式（即技能）。实验表明，Skill DT不仅能执行离线状态边际匹配（SMM），还能发现易于采样的描述性行为。此外，我们证明，通过纯无奖励优化，Skill DT在D4RL基准测试中仍能与监督式离线RL方法相媲美。代码与演示视频见项目页面：https://github.com/shyamsn97/skill-dt