Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process. We trace this instability to a fundamental connection: the flow rollout is algebraically equivalent to a residual recurrent computation, making it susceptible to the same vanishing and exploding gradients as RNNs. To address this, we reparameterize the velocity network using principles from modern sequential models, introducing two stable architectures: Flow-G, which incorporates a gated velocity, and Flow-T, which utilizes a decoded velocity. We then develop a practical SAC-based algorithm, enabled by a noise-augmented rollout, that facilitates direct end-to-end training of these policies. Our approach supports both from-scratch and offline-to-online learning and achieves state-of-the-art performance on continuous control and robotic manipulation benchmarks, eliminating the need for common workarounds like policy distillation or surrogate objectives.

翻译：利用离策略强化学习训练表达能力强的流式策略，由于多步动作采样过程中的梯度病态问题而极不稳定。我们将这种不稳定性追溯到一个根本关联：流式展开在代数上等价于残差循环计算，使其与RNN一样易受梯度消失和爆炸的影响。为解决此问题，我们借鉴现代序列模型原理对速度网络进行重参数化，提出两种稳定架构：Flow-G（采用门控速度）和Flow-T（采用解码速度）。随后，我们开发了一种基于SAC的实用算法，通过噪声增强展开实现对这些策略的直接端到端训练。该方法同时支持从零开始学习与离线到在线学习，并在连续控制和机器人操作基准测试中达到最先进性能，无需依赖策略蒸馏或代理目标等常见变通方案。