We propose ReinFlow, a simple yet effective online reinforcement learning (RL) framework that fine-tunes a family of flow matching policies for continuous robotic control. Derived from rigorous RL theory, ReinFlow injects learnable noise into a flow policy's deterministic path, converting the flow into a discrete-time Markov Process for exact and straightforward likelihood computation. This conversion facilitates exploration and ensures training stability, enabling ReinFlow to fine-tune diverse flow model variants, including Rectified Flow [35] and Shortcut Models [19], particularly at very few or even one denoising step. We benchmark ReinFlow in representative locomotion and manipulation tasks, including long-horizon planning with visual input and sparse reward. The episode reward of Rectified Flow policies obtained an average net growth of 135.36% after fine-tuning in challenging legged locomotion tasks while saving denoising steps and 82.63% of wall time compared to state-of-the-art diffusion RL fine-tuning method DPPO [43]. The success rate of the Shortcut Model policies in state and visual manipulation tasks achieved an average net increase of 40.34% after fine-tuning with ReinFlow at four or even one denoising step, whose performance is comparable to fine-tuned DDIM policies while saving computation time for an average of 23.20%. Project webpage: https://reinflow.github.io/

翻译：本文提出ReinFlow，一种简洁高效的在线强化学习框架，用于微调连续机器人控制中的流匹配策略族。基于严格的强化学习理论推导，ReinFlow通过向确定性流策略路径中注入可学习噪声，将流转换为离散时间马尔可夫过程，从而实现精确且直观的似然计算。该转换机制不仅促进策略探索，还保障训练稳定性，使得ReinFlow能够对多种流模型变体进行微调，包括Rectified Flow [35]和Shortcut Models [19]，特别是在极少数（甚至单步）去噪步骤下仍保持高效。我们在具代表性的运动控制与操作任务中评估ReinFlow，涵盖视觉输入下的长程规划与稀疏奖励场景。在挑战性足式运动任务中，经微调后的Rectified Flow策略片段奖励平均净增长达135.36%，同时相较于前沿扩散强化学习微调方法DPPO [43]节省了去噪步骤与82.63%的墙钟时间。在状态与视觉操作任务中，Shortcut Models策略经ReinFlow在四步甚至单步去噪条件下微调后，成功率平均净提升40.34%，其性能与微调后的DDIM策略相当，同时平均节省23.20%的计算时间。项目页面：https://reinflow.github.io/