Offline goal-conditioned reinforcement learning (GCRL) is a promising approach for pretraining generalist policies on large datasets of reward-free trajectories, akin to the self-supervised objectives used to train foundation models for computer vision and natural language processing. However, scaling GCRL to longer horizons remains challenging due to the combination of sparse rewards and discounting, which obscures the comparative advantages of primitive actions with respect to distant goals. Hierarchical RL methods achieve strong empirical results on long-horizon goal-reaching tasks, but their reliance on modular, timescale-specific policies and subgoal generation introduces significant additional complexity and hinders scaling to high-dimensional goal spaces. In this work, we introduce an algorithm to train a flat (non-hierarchical) goal-conditioned policy by bootstrapping on subgoal-conditioned policies with advantage-weighted importance sampling. Our approach eliminates the need for a generative model over the (sub)goal space, which we find is key for scaling to high-dimensional control in large state spaces. We further show that existing hierarchical and bootstrapping-based approaches correspond to specific design choices within our derivation. Across a comprehensive suite of state- and pixel-based locomotion and manipulation benchmarks, our method matches or surpasses state-of-the-art offline GCRL algorithms and scales to complex, long-horizon tasks where prior approaches fail. Project page: https://johnlyzhou.github.io/saw/

翻译：离线目标条件强化学习是一种在大规模无奖励轨迹数据集上预训练通用策略的有前景方法，类似于训练计算机视觉和自然语言处理基础模型所使用的自监督目标。然而，由于稀疏奖励与折扣因子的共同作用会模糊原始动作相对于远期目标的比较优势，将目标条件强化学习扩展到更长决策时域仍面临挑战。分层强化学习方法在长时域目标达成任务上取得了显著的实证结果，但其对模块化、时域特定策略及子目标生成的依赖引入了显著的额外复杂性，阻碍了向高维目标空间的扩展。本研究提出一种通过基于优势加权重要性采样的子目标条件策略自举来训练扁平化（非分层）目标条件策略的算法。该方法消除了对（子）目标空间生成模型的需求，我们发现这是在大规模状态空间中实现高维控制扩展的关键。我们进一步证明，现有的分层方法与基于自举的方法均对应于本推导框架中的特定设计选择。在一系列全面的基于状态与像素的运动控制与操作基准测试中，本方法达到或超越了当前最先进的离线目标条件强化学习算法，并能扩展到先前方法无法处理的复杂长时域任务。项目页面：https://johnlyzhou.github.io/saw/