Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Diffusion-based policies, while expressive, do not permit conservative likelihood-based updates during fine-tuning because action probabilities are intractable. In contrast, conventional Gaussian policies collapse under multimodality, particularly when actions are executed in chunks, and standard per-step critics fail to align with chunked execution, leading to poor credit assignment. We present SERFN, a sample-efficient off-policy fine-tuning framework with normalizing flow (NF) to address these challenges. The normalizing flow policy yields exact likelihoods for multimodal action chunks, allowing conservative, stable policy updates through likelihood regularization and thereby improving sample efficiency. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy's temporal structure and improving long-horizon credit assignment. To our knowledge, this is the first demonstration of a likelihood-based, multimodal generative policy combined with chunk-level value learning on real robotic hardware. We evaluate SERFN on two challenging dexterous manipulation tasks in the real world: cutting tape with scissors retrieved from a case, and in-hand cube rotation with a palm-down grasp -- both of which require precise, dexterous control over long horizons. On these tasks, SERFN achieves stable, sample-efficient adaptation where standard methods struggle.

翻译：真实世界中灵巧操作策略的微调仍然面临挑战，原因在于有限的实际交互预算和高度多峰的动作分布。基于扩散的策略虽然表达能力丰富，但无法在微调过程中进行保守的似然更新，因为动作概率难以计算。相比之下，传统高斯策略在多峰性下会崩溃，特别是在动作以分块方式执行时，而标准的单步评判器无法与分块执行对齐，导致信用分配不佳。我们提出SERFN——一种结合归一化流（NF）的样本高效离策略微调框架，以应对这些挑战。归一化流策略能为多峰动作分块提供精确的似然，通过似然正则化实现保守且稳定的策略更新，从而提升样本效率。同时，动作分块评判器评估完整的动作序列，使价值估计与策略的时间结构对齐，并改进长时序信用分配。据我们所知，这是首次在真实机器人硬件上展示基于似然的多峰生成策略结合分块级价值学习。我们在真实世界中两个挑战性的灵巧操作任务上评估SERFN：从工具盒中取出剪刀并切割胶带，以及采用手掌朝下握持进行手内立方体旋转——这两项任务都要求长时间跨度上的精确灵巧控制。在这些任务上，SERFN实现了稳定且样本高效的适应性，而标准方法则难以胜任。