Generative models have emerged as a powerful class of policies for offline reinforcement learning (RL) due to their ability to capture complex, multi-modal behaviors. However, existing methods face a stark trade-off: slow, iterative models like diffusion policies are computationally expensive, while fast, single-step models like consistency policies often suffer from degraded performance. In this paper, we demonstrate that it is possible to bridge this gap. The key to moving beyond the limitations of individual methods, we argue, lies in a unifying perspective that views modern generative models, including diffusion, flow matching, and consistency models, as specific instances of learning a continuous-time generative trajectory governed by an Ordinary Differential Equation (ODE). This principled foundation provides a clearer design space for generative policies in RL and allows us to propose Generative Trajectory Policies (GTPs), a new and more general policy paradigm that learns the entire solution map of the underlying ODE. To make this paradigm practical for offline RL, we further introduce two key theoretically principled adaptations. Empirical results demonstrate that GTP achieves state-of-the-art performance on D4RL benchmarks - it significantly outperforms prior generative policies, achieving perfect scores on several notoriously hard AntMaze tasks.

翻译：生成模型因其捕捉复杂多模态行为的能力，已成为离线强化学习中一类强大的策略。然而，现有方法面临一个明显的权衡：扩散策略等缓慢的迭代模型计算成本高昂，而一致性策略等快速的单步模型则往往性能下降。本文证明，弥合这一差距是可能的。我们认为，超越单个方法局限性的关键在于一个统一的视角，该视角将包括扩散模型、流匹配和一致性模型在内的现代生成模型，视为学习由常微分方程控制的连续时间生成轨迹的具体实例。这一原则性基础为强化学习中的生成策略提供了更清晰的设计空间，并使我们能够提出生成轨迹策略——一种学习底层常微分方程完整解映射的全新且更通用的策略范式。为使该范式适用于离线强化学习，我们进一步引入了两个关键的理论原则性适配。实证结果表明，GTP在D4RL基准测试中实现了最先进的性能——它显著优于先前的生成策略，在多个公认困难的AntMaze任务上获得了满分。