Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions. However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering(RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy. We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings when adapting pretrained base policies. Project website:https://weirdlabuw.github.io/rfs.

翻译：模仿学习已成为机器人学中引导序列决策的有效方法，即使在高维灵巧操作任务中也表现出色。近期的行为克隆方法进一步利用扩散模型和流匹配等表达能力强的生成模型来表示多模态动作分布。然而，以此方式预训练的策略通常泛化能力有限，需要在部署时进行额外的微调以实现鲁棒性能。这种适应过程必须保留预训练的全局探索优势，同时能够快速纠正局部执行误差。我们提出残差流引导（RFS），一种用于适应预训练生成策略的数据高效强化学习框架。RFS通过联合优化残差动作和潜在噪声分布来引导预训练的流匹配策略，从而实现互补的探索形式：通过残差校正进行局部优化，通过潜在空间调制进行全局探索。该设计在保留预训练策略表达结构的同时实现了高效适应。我们在灵巧操作任务上验证了RFS的有效性，展示了在仿真和真实场景中适应预训练基础策略时的高效微调能力。项目网站：https://weirdlabuw.github.io/rfs。